1. Establishing the Integrity of Data:
Measurement Systems Analysis
prepared by
Jeffrey T. Luftig, Ph.D.
W. Edwards Deming Professor of Management
Lockheed Martin Engineering Management Program
University of Colorado at Boulder
Copyright 2002 Luftig & Warren International

2. Topics Measurement Scales and Types of Data Establishing the
Reliability and Validity of Instrumentation, or
Precision and Accuracy of Instrumentation
Copyright 2002 Luftig & Warren International

3. Measurement Scales and Data
Copyright 2002 Luftig & Warren International

4. Measurement Scales and Data
Copyright 2002 Luftig & Warren International

5. Measurement as a Process
As in any process, regardless of the nature of data collected or generated, measurement systems must demonstrate
Stability through time, or control
Minimal variation as a proportion of specifications, or capability
Minimal variation as a proportion of process variation
Copyright 2002 Luftig & Warren International

6. Measurement as a Process
MEASUREMENT PROCESS
Equipment
Standard
Procedure
Operator
Measurement
Ambient
Environmental Characteristics
Product or Process to be Measured
Copyright 2002 Luftig & Warren International

7. Definition of Terms Reference Value Resolution
The theoretically or agreed upon correct value of the characteristic being measured, traceable to some standard
Resolution
The smallest increment, or unit of measure, available from a measurement process
Generally at least 1/10th of the specification range
Copyright 2002 Luftig & Warren International

8. Definition of Terms Precision Accuracy (Bias)
The degree of agreement (or variability) between individual measurements or test results from measuring the same specimen(s)
Accuracy (Bias)
The difference between the average of the measurement error distribution and the reference value of the specimen measured
Copyright 2002 Luftig & Warren International

9. Precision vs. Accuracy Precision Accuracy
Copyright 2002 Luftig & Warren International

10. Definition of Terms Repeatability Reproducibility
The variation in repeated measurements of the same items with a single measurement system
Within appraiser/system variation
Reproducibility
The variation in the average measurements by different appraisers or systems measuring the same items
Between appraiser/system variation
Copyright 2002 Luftig & Warren International

11. Measurement Error Distribution of repeated measures on a single
specimen or part
Precision
- Repeatability
- Reproducibility
Accuracy
(Bias)
Reference Value
Copyright 2002 Luftig & Warren International

12. Terms Linearity Stability
The degree to which bias changes with changes in the magnitude of the characteristic measured
Stability
The dependability, or consistency of the measurement process over time
Copyright 2002 Luftig & Warren International

13. Measurement Systems Capability
The variability resulting from measurement error must not exceed a significant proportion of the intended specifications said to be capable
In addition, it is not desirable for measurement error to exceed a significant proportion of the total process variability
Capability is not the same as acceptability, acceptability must be determined on a case by case basis
Copyright 2002 Luftig & Warren International

14. Measurement Systems Capability
LSL
USL
5.15E
(USL - LSL)
Measurement
Error Distribution
Copyright 2002 Luftig & Warren International

15. Measurement System Studies
Potential Studies
Assess potential of a measurement system to be capable over the long term
10 parts measured 2–3 times by one or more appraisers
A “quick and dirty” study to find out if you are in the ballpark
Assesses repeatability and reproducibility
Often called an R&R study
Copyright 2002 Luftig & Warren International

16. Measurement System Studies
Potential Studies
Error Through Time
Bias Through Time
Copyright 2002 Luftig & Warren International

17. Measurement System Studies
Potential Studies (continued)
Copyright 2002 Luftig & Warren International

18. Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System
This result of the previous analysis allows us to calculate the average variance of the repeated measures, which when we take its square root gives us the estimate of the standard deviation due to measurement error:
2 = 66.39
 =
Copyright 2002 Luftig & Warren International

19. Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System
Using the estimate of measurement error, we can calculate the Precision-Tolerance ratio, which in the case of short-term studies, should be less than 10%. Assuming the engineering tolerance for this process is 470 (USL) – 450(LSL) = 20:
2 = 66.39
 =
P/T = Precision-Tolerance Ratio =
= 6() / USL-LSL
= 6(8.15) / 470 – 450
= 2.44
= 244% > 10% Requirement (S-T)
Copyright 2002 Luftig & Warren International

20. Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System
Likewise, we can estimate the Accuracy (amount of Bias) in the scale by calculating the average of the differences between the Means of the Repeated Measures and the True Values for the associated specimens:
 = -3.09
Estimate Bias at 3.09 Grams; as
compared to the Precision estimate,
this is arguably an inconsequential
value.
Copyright 2002 Luftig & Warren International

21. Measurement System Studies
Short-term Studies
25 parts measured 5-8 times by one or more appraisers
A more thorough short-term assessment
Long-term Studies
8-10 parts measured 25+ times by one or more appraisers
Assesses through time stability
Copyright 2002 Luftig & Warren International

22. Measurement System Studies
Long-term Studies
Copyright 2002 Luftig & Warren International

23. Measurement System Studies
Long-term Studies
Copyright 2002 Luftig & Warren International

24. Measurement Systems Requirements
Summary
Regardless of the type of data gathered by an instrument, and the assessment methodology employed, the instrument or device utilized to obtain criterion data must meet three requirements before the experiment should proceed:
The instrument must be precise or reliable;
The instrument must be accurate or valid; and
The instrument should be / must be operating in a state of statistical control.
Copyright 2002 Luftig & Warren International

25. Sources and References
The material used in the PowerPoint presentations associated with this course was drawn from a number of sources. Specifically, much of the content included was adopted or adapted from the following previously-published material:
Luftig, J. A Quality Improvement Strategy for Critical Product and Process Characteristics. Luftig & Associates, Inc. Farmington Hills, MI, 1991
Spooner-Jordan, V. Understanding Variation. Luftig & Warren International, Southfield, MI 1996
Luftig, J. and Petrovich, M. Quality with Confidence in Manufacturing. SPSS, Inc. Chicago, IL 1997
Littlejohn, R., Ouellette, S., & Petrovich, M. Black Belt Business Improvement Specialist Training, Luftig & Warren International, 2000
Ouellette, S. Six Sigma Champion Training, ROI Alliance, LLC & Luftig & Warren, International, Southfield, MI 2005
Luftig, J. An Overview of Total Quality Management, Luftig & Warren, International, 1992
Luftig, J. Dr. Deming’s Theory of Profound Knowledge as a Foundation for Strategic Planning and Policy Deployment, Luftig & Warren, International, 1997
Luftig, J. and Jordan, V. Design of Experiments in Quality Engineering, McGraw-Hill/Irwin Publishing Company, 1998
Copyright 2002 Luftig & Warren International