LESSON 7 Data Collection and Analysis

LESSON 7 Data Collection and Analysis
15November2013 Lesson 4: Safety Stock

Lesson Introduction Given a Government Contract Quality Assurance (GCQA) surveillance plan, the student will be able to develop and execute a Data Collection and Analysis (DC&A) plan via the data collection and analysis optional checksheet.

Lesson Objectives Upon completion of this lesson, you should be able to: Relate the importance of data collection and analysis to Defense Contract Management Agency (DCMA) Quality Assurance (QA) responsibilities. Determine the type of data for collection and evaluation. Determine the source of the data identified for collection. Distinguish the difference between Attribute Data and Variable Data. Determine evaluation intervals for collected data. Lesson 4: Safety Stock

Lesson Objectives (cont.)
Upon completion of this lesson, you should be able to: Determine the evaluation tool necessary to analyze performance data. Examine data analysis results for trends in quality. Relate the analysis of collected data to the adjustment of GCQA surveillance. Identify the minimum documentation requirements for data analysis results and actions taken. Lesson 4: Safety Stock

Lesson Topics This lesson covers the following topics: GCQA Authority
GCQA Strategy Data Collection and Analysis Plan Form Data Analysis Levels and Traceability Data Analysis Documentation Requirements

WIIFM? This lesson will:
Identify the DCMA policy for performing data analysis. List the questions to ask yourself before beginning data analysis. Define which types of data evaluate what acquisition participants. Provide data evaluation methods and which are the best to use with different types of data. Introduce the data analysis check list.

GCQA Authority Lesson Topics: GCQA Authority GCQA Strategy
Data Collection and Analysis Plan Form Data Analysis Levels and Traceability Data Analysis Documentation Requirements

Topic 1: GCQA Authority Federal Acquisition Regulation (FAR) , Contract Administration Office (CAO) Responsibilities Defense Federal Acquisition Regulation Supplement (DFARS) , Quality Evaluation Data DFARS Procedures, Guidance, and Information (PGI) , Quality Evaluation Data

FAR 46.104, CAO Responsibilities
When a contract is assigned for administration to the CAO: Develop and apply efficient procedures for performing Government contract quality assurance actions under the contract in accordance with the written direction of the contracting office Perform all actions necessary to verify whether the supplies or services conform to contract quality requirements Maintain, as part of the performance records of the contract, suitable records …

DFARS , QA Policy Departments and agencies shall develop and manage a systematic, cost-effective Government contract quality assurance program to ensure that contract performance conforms to specified requirements.

DFARS 246.470-2 Quality Evaluation Data
The CAO shall establish a system for the collection, evaluation, and use of the types of quality evaluation data specified in DFARS PGI

DFARS PGI 246.470-2, Quality Evaluation Data
Types of quality evaluation data are: Quality data developed by the supplier during performance Data developed by the Government through contract quality assurance actions Reports by users and customers

Responsibilities FAR 46.104, CAO Responsibilities
DFARS , Quality Evaluation Data DFARS PGI , Quality Evaluation Data Data Collection and Analysis - QA Policy Agency implementation policy

Question and Answer (Select all that apply.)
According to FAR , the CAO is responsible for ___________. Applying efficient procedures for performing Government contract quality assurance actions Maintaining surveillance records Ordering stop production on non-acceptance of product Performing all actions ensure conformance to contract quality requirements

Question and Answer Which policy mandates that DCMA have a QA program that provides for systematic and cost effective determination that contract requirements have been met? FAR DFARS DFARS PGI

Question and Answer What are the types of quality evaluation data used in data analysis? Inspection, witness, verification Product examination, process review, system audit Supplier, Government, customer Documentation, systematic, procedural

GCQA Strategy Lesson Topics: GCQA Authority GCQA Strategy
Data Collection and Analysis Plan Form Data Analysis Levels and Traceability Data Analysis Documentation Requirements

Topic 2: GCQA Strategy Contract Technical Review (CTR) Risk Assessment
GCQA Surveillance Planning Perform Surveillance Product Release SOW Tech. Item System Req. Schedule/Cost Exhibits Data Pkg. Impact Performance Causes Likelihood QA Surveillance Methods System Audit Process Review Product Exams Data Collection and Analysis (DC&A) is part of the Plan, Do, Check, Act (PDCA) cycle. DC&A Records

Why Do It Provides basis of confidence for product acceptance
Determines acceptability and effectiveness of supplier’s quality system Provides basis for decisions to increase, reduce, or re-focus supplier’s facility surveillance Provides customer with objective evidence of supplier’s ability to produce conforming product Provides basis for decisions related to the allocation of Government resources Provides probability or likelihood component of QAS’s risk assessments

Data Collection Planning (1 of 3)
QAS plans data collection and analysis (DC&A) DC&A planning included in surveillance plan and addresses Data to be collected Collection and analysis method and interval PROCESS (Potential Risk Cause) DATA SOURCE OF DATA DCMA, Supplier, Customer DATA OWNER Quality, Engineering, Purchasing, Production TYPE OF DATA Attribute or Variable INTERVAL OF COLLECTION Daily, Weekly, Monthly, Quarterly ANALYSIS TOOL (Method) Run Chart, Pareto, Histogram, Check Sheet INTERVAL OF ANALYSIS Daily, Weekly, Monthly, Quarterly, Semi-Annual, Annual

Questions answered: Trends over time? Effectiveness of supplier inspection over time? How well the process is performing over time? Health/adequacy of the Quality Management System (QMS)? Decide on the type of data to be collected Attribute Variable

Determine the frequency/interval for availability and collection When, what, where, and from who is the data available? Decide on the method (tools) used for evaluation of each type of data Decide on analysis interval What is the supplier’s throughput? How many units do they ship per month?

Question and Answer Why is data analysis important? (Select all that apply.) Provides basis of confidence for product acceptance Provides basis for decisions related to the allocation of Government resources Provides probability or likelihood component of risk assessments Provides system for evaluation and use of data types

Data Collection and Analysis Plan Form
Lesson Topics: GCQA Authority GCQA Strategy Data Collection and Analysis Plan Form Data Analysis Levels and Traceability Data Analysis Documentation Requirements

Topic 3: Data Collection and Analysis Plan Form
DC&A plan includes: Process selection Data sources Evaluation tools ECPs/ECNs In-process Inspection & Test Control Charts

Process Selection Processes were identified during the risk assessment phase. Identify those processes in the planning document. Ask the questions: “What do I need to know about these processes?” “What data will provide that information?” “Is the data produced by the supplier, DCMA, or provided by the customer, or a mix of several data sources?” “What functional group provides that information?” “Is it variable or attribute data?”

Final Test/Inspection Results
Supplier Data Sources Use-As-Is Pareto Source Inspection Data % Defective Defects Per Unit Final Test/Inspection Results Internal Corrective Actions/CARs Waivers Process Yield Deviations ECPs/ECNs Incoming Inspection Cost Related to Quality Third Party Audits Repairs Machine Process Monitoring Defects Per Opportunity In-process Inspection & Test Rework Material Review Board (MRB) Standard Repairs Internal Quality Audits Control Charts

DCMA/Customer/User Data Sources
Customer Audits Quality System Audits DCMA Forum Customer Complaints DRs/PQDRs Production Surveillance LoD In-process Product Examinations/Tests DCMA CARs GIDEP Release Quality QALI First Article Results Field Failure Reports Customer Reviews Vendor Alerts Pre-award Surveys Process Reviews Final Product Examinations/Tests Post Award Orientation Conferences Host Nation QA Packing & Packaging Discrepancies Positive Customer Feedback

Selecting Evaluation Tools
Arranging data in graphs Helps “crack the code” Better reveals information hidden in the data Determine best graph(s) for data depends on: Data type collected Graph’s purpose

Selecting Evaluation Tools (cont.)
Decide on the method (tools) used for evaluation of each type of data For trend analysis (Run Chart): Process Yield Defects/Defects per unit/Defectives per lot Number of Material Review Board (MRB) actions per time period Scrap Rates For identifying the most significant problems: Pareto Charts (Analysis) Pie Chart Check Sheet

Attribute and Variable Data
ATTRIBUTE DATA - A result such as: Pass or Fail Go or No Go Good or Bad VARIABLE DATA - Actual measure such as: Temperature Time Inches Decibels

Analysis Tool Examples
Methods (tools) for analysis: Check sheets Run Charts Pareto Charts Pie Charts Control Charts [Statistical Process Control (SPC)]

Question and Answer During DCMA production lot testing, the concentricity measurement of of diameter A was found to be out of tolerance. What type of data is this? (Select all that apply.) Attribute Variable Government Customer

Question and Answer What is the best chart to use to track how many defects per unit or lot over time? Control Chart Run Chart Pie Chart Pareto Chart

Check Sheet What is it? Uses
A simple tool for recording data and to determine how often a defect occurs. Usually the first step in recording data so it can be further analyzed. Uses Stratify (group) data Prove or disprove feelings (opinions) Identify numbers of defects Identify types of defects

Check Sheet Example Defect Type Totals Customer Complaints Totals II
Assembly II 2 Print Quality IIIIIIIIIIIII 13 Print Detail IIII 4 Edge Flaw IIIIIIIIIIIIIIIIIIIIII 22 Cosmetic IIIII 5 Customer Complaints Totals Missing Ring II 2 Print Quality IIIIIIIIIIIIIIIIIIIIIII 23 Misplace Print IIII 4 Rough Edge III 3 Type Error IIIIII 6 Excess Flash IIIIIIIIIIIII 13 Late Shipment Bad Count

Run Chart What is it? Uses
A visual representation of process performance or activity over a specific period of time (can be by lot or unit) Very good for identifying trends! Uses Tracking scrap rates Tracking the number of MRB actions Tracking the number of rejects per lot Tracking the number of defects per unit Process Yield

Run Chart Example (1 of 3) Data Source: B&B Printing, Final Print Inspection Process Reports (Contractor) Report Range: January through March 2011 Chart Created on: 4/15/2011

Run Chart Example (2 of 3) Data Source: QAS Product Examination Results at Final Print Inspection Report Range: January through March 2011 Chart Created on: 4/15/2011

Run Chart Example (3 of 3) Data Source: Contractor Yield and QAS Product Examination Results at Final Print Inspection Report Range: 1/2011 through 3/2011 Chart Created on: 4/15/2011

Pareto Chart What is it? Uses
Vertical bar graph used to identify relative importance of nonconformances Uses Rank the significance of problems by: Scope, frequency, or number of occurrences Cost, time, or distance Criticality, urgency, or visibility Examine issues using more than one variable (number, cost, time, etc.). Look for classic split

Pareto Chart Example (1 of 3)

Pie Chart What is it? Uses
Circular graph that represents 100% of the displayed data. Pie “slices” that show the relative share of the data. Uses Rank the significance of problems by: Scope, frequency, or number of occurrences Cost, time, or distance Criticality, urgency, or visibility Examine issues using more than one variable (number, cost, time, etc.). Look for classic split Easier to understand than Pareto chart

Pie Chart Example

Question and Answer Which is the best graph to use to identify trends?
Pareto Chart Pie Chart Check Sheet Run Chart

Question and Answer Which is the best graph to use to identify relative importance of nonconformances? Pareto Chart Pie Chart Check Sheet Run Chart

CONTROL CHART SELECTION TOOL
Attribute Data Variable Data Displays Constant Sample Size Variable Sample Size Defectives NP-Chart P-Chart X-bar and R Chart Defects C-Chart U-Chart

Attribute Data Control Charts
Four types of charts for attribute data: P-Chart: Proportion (percentage) defective with a varying sample size NP-Chart: Number of defectives with a constant sample size C-Chart: Number of defects (nonconformances) per sample with a constant sample size U-Chart: Number of defects with a varying sample size

Variable Data Control Charts
Two charts used for collected variable data. (X-bar) provides information concerning process average R provides information as to how variable the data is with regard to time Together, X-bar and R charts indicate stability and uniformity of the process outputs.

Process Capability Process capability identified by Cp
Cp assumes process mean is centered on specification nominal value or target Determine if process is capable of producing product that meets engineering specification after determining that process producing “variable data” is in statistical control Establish a process capability for critical safety items (CSI) characteristics/features Determine capability by comparing engineering tolerance upper specification limit/lower specification limit (USL/LSL) to process performance spread/range

Cp < 1 Cp > 1.33 Cp ≥ 2.0 Cp Interpretation Cp = 1-1.33
Process is NOT capable of meeting specification over time Cp < 1 Process is marginally capable of meeting specification over time Cp = Process is capable of meeting specification over time Cp > 1.33 6 Sigma quality over time Cp ≥ 2.0

Process Capability Index
Process capability index identified by Cpk Measures how close process mean is running to engineering USL and LSL Calculate Cpk when process performance (mean) is not centered on nominal/target value of engineering specification (X-bar and R charts) Process mean drifts either direction of nominal value Left of nominal Cpl (lower) and right of nominal Cpu (upper) Cpk is smaller of the two (worst case)

Cpk Interpretation Cpk < 1 Cpk = 1-1.33 Cpk > 1.33 Cpk ≥ 1.5
Process is NOT meeting specification over time Cpk < 1 Process is marginally meeting specification over time Cpk = Process is meeting specification over time Cpk > 1.33 6 Sigma quality over time Cpk ≥ 1.5 When making decisions on adjusting surveillance, the Process Capability Index (Cpk) is a more valuable figure to use, especially for mature processes.

Cp Interrelationship Cpk

Cp Interrelationship Cpk (cont.)
Cp does not change when the process mean changes Cp = Cpk when the process performance mean is exactly the same as the engineering specification nominal (target) value A Cpk of zero means that the process performance mean is equal to either the LSL or USL A Cpk can be a negative value only when the process performance mean is outside either the LSL or USL

Control Chart Construction
Collect 20 – 25 groups of samples prior to calculating the statistics and control limits Consider using historical data to set baseline Calculate statistically, upper and lower control limits… don’t confuse with specification limits Use correct chart for data type Record data in same sequence it was collected; otherwise control chart is meaningless Data must reflect how the process performs “naturally.” Do not make adjustments to the process while collecting data, other than that allowed by procedures. This is referred to as “tampering.”

Data Collection and Analysis Plan
Surveillance plan shall include plan for DC&A and address the following minimum information: Process Data to be collected Source of Data Owner of Data Data Type Intervals of collection Method of analysis Intervals of analysis

Question and Answer What two charts are used for variable data? (Select all that apply.) X-bar Pareto R Capability process

Question and Answer What does this bell curve represent?
Positive supplier process capability Neutral supplier process capability Negative supplier process capability

Question and Answer What information does the following graph display?
Process is in control; process capability index is within limits Process is out of control; process capability index is outside of upper specification limits Process is out of control; process capability index is within limits Process is in control; process capability index is outside of upper specification limits

Interval Determination
Evaluation and data analysis interval must take into account data type and volume to be reviewed, as well as impact on product quality (risk); could be a result of failure to identify adverse trends in a timely manner. Recommended intervals: Monthly for resident facilities Quarterly for non-resident facilities Mandatory intervals: Every six months for active suppliers Annually for infrequent suppliers

Data Collection Requirements
Collect the data in accordance with DC&A plan. Data collection does not require separate/duplicate file copy maintenance for specific purpose of evaluation. However, it may be advantageous to maintain certain records for a continuous review of trends. Treat supplier data as For Official Use Only (FOUO) and appropriately protect to prevent unauthorized access or disclosure. FOUO (FOUO marking for illustration/training purposes only)

Exercise: Develop DC&A Plan
Students work in pairs to develop/document a DC&A plan in the DC&A Plan tab of the Risk Profile and Plan Tool. One student open the Risk Profile and Plan Tool saved in the Module 4, Lesson 5 exercise. One student open the following documents: Contractor_Performance_History.docx Contractor_Performance_History_DCnA.docx Valley Forgings_SystemsAuditReport.docx ValleyForgings_ProductExamSheet_1711.xlsx ValleyForgings_SystemsAudit_FindingsAttachment.pdf Time: 20 minutes

Data Analysis Levels and Traceability

Topic 4: Data Analysis Levels and Traceability
DC&A methodology ranges from: QMS-Level - review of all performance data sources [i.e., First Pass Yield (FPY), process reviews (PRs), product examinations (PEs), etc.] to guide QAS to specific areas of weakness in the supplier’s QMS Key/critical process/characteristic-level - detailed review relating to a specific process/characteristic Valid customer complaints traceable to a deficiency in the supplier’s operation indicate the need for risk reassessment and surveillance strategy revision (intensify) in that area to preclude the shipment of defective material.

Data Analysis Data Evaluation (analysis) Trends
Health of the supplier’s quality effort Repetitive and overlooked deficiencies Process stability and capability

Risk Assessment accomplished
Has Risk Changed? If risk has changed, the surveillance strategy must change (and be documented in the GCQA Surveillance Plan) to be commensurate with the risk Risk Assessment accomplished If significant change in system, process, or characteristic performance, revise risk assessment based on likelihood

Surveillance Plan Portion
Risk Change Example Supplier process analyzed is Acceptance Testing, rated Low Risk: Original strategy: product examination – verification (test data review) at final inspection; initial process review of acceptance test process Past six months DC&A results indicate two valid Product Quality Deficiency Reports (PQDRs) for product delivered with characteristics out of specification Risk reassessment results: Supplier’s process now Moderate Risk GCQA surveillance strategy adjusted to witnessing one unit per day; add a process review of the assembly process (product examination – witness testing) Surveillance Plan Portion

The New Likelihood Table
Risk Cause Likelihood How Likely is the Potential Risk Cause to Occur? High It is highly likely to occur. Performance data shows evidence of an inability to meet the contractual requirements. The process is extremely difficult to perform. Moderate It is probable or likely to occur. No data available to show the Supplier’s ability to meet contractual requirements. The process is somewhat difficult to perform. Low It is unlikely that the risk will occur. Performance data shows evidence that the contractual requirements will be met. It is a common process and not difficult to perform. High Number of Nonconformances/Low FPY and/or Cp/Cpk = 1.00 or Less State-of-the-Art/ Technology Maturity Inconclusive Considerable Process Variance - Cp/Cpk = 1.00 – 1.32 or Control Points Positive Trends Normal Process Variance (Control Points) or Capable (Cp/Cpk = 1.33 or Higher {Better})

GCQA Adjustments and Actions
Customer Input & Outcomes QAS Surveillance Data Supplier Data Data Analysis Adjustments Create analysis record Actions Revise risk Change GCQA surveillance (intensity/frequency/method) Modification to delegations Implement/Withdraw Certificate of Conformance (CoC)/Alternative Release Procedures (ARP) Issue CAR Add new processes/characteristics Update GCQA surveillance plan Request relief from mandatory inspections [Quality Assurance Letter of Instruction (QALI)/Letter of Delegation (LOD)]

Adjusting GCQA Surveillance
What changes if risk changes? Scope + Frequency + Intensity + Technique Inspect System Audit Process Review Witness Product Examination Test Verify “Goal” Maintain a High Confidence

Data Analysis Documentation Requirements

Topic 5: Data Analysis Documentation Requirements
QA personnel shall record results of data analysis, (electronically or on paper); minimum documentation requirements: Date of analysis Individual performing the analysis Data analyzed Results of the analysis (conclusions) Actions taken as a result of the analysis

Results of Analysis What conclusions have I drawn from the analysis?
What did the data tell me? Is the process getting better, worse, or has it remained the same?

Update surveillance plan
Action Taken Action(s) taken Intensify surveillance Reduce surveillance Focus surveillance in other areas Adjust checklists Modify your data analysis plan Update surveillance plan

Defect Analysis Example
Data Source: Contractor Yield and QAS Product Examination Results at Final Print Inspection Report Range: 1/2011 thru 3/2011 Chart Created on: 4/15/2011

Adequacy Analysis Example
Date Period Covered Process(es) DATA ANALYZED RESULTS OF ANALYSIS (Update Performance Factors) ACTIONS PLANNED/TAKEN (Update RP&P) Government Data Supplier Data User Data 19-Jul-10 3rd Quarter 2010 Product Marking CAR 0FL ; Inspection records Inspection records N/A 19 Jul 10-ACOG P/N TA31RCOM150, S/N , failed the UID reading. The scope in question had to be remarked. Four elements of the reading received a D or F rating. This was an isolated incident. During product acceptance, the next lot of material will use a .15% AQL. (Subsequent inspections have not detected this defect). No further action is required since this defect can be fixed on the spot. 16-Aug-10 Assembly CAR 0FL ; Inspection records N/A 16 Aug 10- P/N TA684MDO, Medium Machine Gun Day Optic, S/N was presented for government inspection without the anti reflective device, and both the front and rear flip up protective caps. This was an isolated incident. Subsequent inspections have not detected this defect. Three of the five CARs were in the assembly process. Therefore, the assembly process has been scheduled for review and proofing in January 2011. 10-Sep-10 CAR 0FL ; Inspection records N/A 10 Sep During final inspection (P/N TA31RCOM150, NSN , Sight Bore Optical), it was discovered that one of the optic sights red cross hairs did not emit a red glow. This was cause for rejection of the entire lot of 1520 optic sights. This was an isolated incident. Subsequent inspections have not detected this defect.

Pitfalls to Avoid Lack of proper planning … no formal DCMA DC&A plan of attack Lack of evaluation of the supplier’s data collection process (data integrity) Analysis performed on parts rather than on the process Part numbers instead of processes Not adjusting surveillance strategy when data shows processes are in control Supplier not performing Root Cause Analysis (RCA) Managing defects instead of finding their cause Analyzing supplier defect codes that are vague and/or misleading Untimely review of the data No duplication of the data analysis results

Summary Having completed this lesson, you should understand:
Data collection and analysis authorized by FAR , DFARS , and DCMA policy. Data from suppliers, Government, and customers used to perform data analysis. Attribute data is a result such as pass or fail, go or no go, or good or bad. Variable data is an actual measurement such as temperature, time, inches, or decibels. Evaluation and data analysis intervals must take into account the data type and volume, as well as the risk, which could result in a failure to identify adverse trends in a timely manner. Lesson 4: Safety Stock

Summary (cont.) Having completed this lesson, you should understand:
Many types of graphs or analysis methods available. Choosing the best method depends upon the kind of data collected and the collection purpose. If risk has changed, surveillance strategy must change and be documented in the GCQA surveillance plan. Scope, intensity, frequency, and method can be changed or adjusted in the GCQA surveillance plan as a result of data analysis. Data analysis records shall identify, as a minimum, the analysis date, individual performing the analysis, data analyzed, analysis results (conclusions), and any actions taken as a result of the analysis. Lesson 4: Safety Stock

Questions

Review Question 1 Government, supplier, and customer data are types of data used for evaluation according to _______________. FAR DFARS DFARS DFARS PGI

Review Question 2 What is the most efficient way to obtain a basis of confidence for product acceptance and determine acceptability of a supplier’s quality system? Product examination Process review Data analysis System audit

Review Question 3 While the QAS is reviewing supplier inspection notes, a measurement is found to be nonconforming. What type of data is this? (Select all that apply.) Attribute Government Supplier Variable

Review Question 4 Which evaluation method is appropriate for identifying issues using more than one variable? Check sheet Pareto chart Run chart Control chart

Review Question 5 Which evaluation method uses both X-bar and R charts to determine if the process is capable? Check sheet Pareto chart Run chart Control chart

Review Question 6 Which evaluation method is the easiest and most basic method to record data to analyze? Check sheet Pareto chart Run chart Control chart

Exercise: DC&A Students work in pairs using provided customer, supplier, and DCMA data to determine appropriate surveillance adjustments to the surveillance plan developed in Module 4, Lesson 5. Required materials: Contractor_Performance_History.docx Contractor_Performance_History_DCnA.docx ValleyForgings_SystemAuditReport.docx ValleyForgings_SystemsAudit_FindingsAttachment.pdf ValleyForgings_ProductExamSheet_1711.xlsx Surveillance Plan (from Module 4, Lesson 5) Be ready to discuss the adjustments as a class. Time: 20 minutes

Case Study: Part 4 Analysis, Corrections, and Release
Students work in pairs to resolve Case Study Situation(s). Use the Case Study: Part 4 – Subcontractors and Delegation document: CMQ101_M6_CaseStudy.pdf Class review/discussion upon completion. Time: 45 minutes

LESSON 7 Data Collection and Analysis

Similar presentations

Presentation on theme: "LESSON 7 Data Collection and Analysis"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

LESSON 7 Data Collection and Analysis

Similar presentations

Presentation on theme: "LESSON 7 Data Collection and Analysis"— Presentation transcript:

Similar presentations

About project

Feedback