1. A Pragmatic Method for Pass Fail Conformance Reporting
A general approach for meeting global requirements
Agilent Technologies
Michael Dobbert (presenter)
Robert Stern
What are the criteria for stating Pass/Fail conformance when calibrating an instrument and comparing the measured results against specifications? 39 Agilent Calibration Laboratories (Service Centers) in 26 countries audited by 14 different accreditation bodies. It is valuable to draw attention to the different criteria and the rational behind them.

2. Addressing Global Requirements
Upper Tolerance
Case 1
Case 2
Case 3
Case 4
ISO/IEC 17025
ANSI/NCSL Z540-1
When statements of compliance are made, the uncertainty of measurement shall be taken into account.
Where calibrations provide for verification that measurement quantities are within specified tolerances, the probability that incorrect acceptance decisions (false accept) will result from calibration test shall not exceed 2%...
ILAC-G8
Purpose of standards documents: metrology quality. Each document gives different criteria for pass/fail judgments.
All attempt to control the risk of false accepts. Highlight the differences between Z540.3 and ILAC-G8.
ANSI/NCSL Z540.3
EURAMET/cg-15/v.01
(previously EA 04/13)
Subsequent to calibration and under normal conditions of use, the uncertainty associated with the readings of a DMM will be the combination of the DMM’s specification and the calibration uncertainty.

3. The Significance of Measurement Error
y = x + e (observed value)
Explain the graph. Gaussian assumptions. Process for generating the graph.
Observed Value: true value offset by the measurement error
x (true value)

4. y = x + e (observed value)
False Accept Risk
tolerance limit
tolerance limit
acceptance limit
False accept
y = x + e (observed value)
False accept
acceptance limit
False accept risk: percentage of points in the false accept corners relative to the total population.
Acceptance limits are process control limits: applying these acceptance limits to a known population of device results in a predictable number of false accepts over time.
Guard band can be set to achieve a particular false accept risk.
x (true value)

5. y = x + e (observed value)
ILAC-G8 Guard Band
Fail
Indeterminate
Pass
Indeterminate: Compliance or non-compliance cannot be stated at a 95% level of confidence.
tolerance limit
acceptance limit
95% Expanded Uncertainty
y = x + e (observed value)
False accept
Restate what the x-axis and y-axis are.
Common misconception: This means that the probability that the measurement is below the upper specification
limit is 95 %, i.e. approximately 97.5 % for symmetrical distributions. (not true)
95% applies only to the measurement uncertainty. I will demonstrate that this is true.
x (true value)

6. ILAC-G8 Guard Band Fail Indeterminate y = x + e (observed value)
95% Expanded Uncertainty
Fail
Indeterminate
Pass
Indeterminate: Compliance or non-compliance cannot be stated at a 95% level of confidence.
tolerance limit
95% Expanded Uncertainty
y = x + e (observed value)
False accept
Restate what the x-axis and y-axis are.
Common misconception: This means that the probability that the measurement is below the upper specification
limit is 95 %, i.e. approximately 97.5 % for symmetrical distributions. (not true)
95% applies only to the measurement uncertainty. I will demonstrate that this is true.
x (true value)

7. Confidence Level versus Probability
tolerance limit
95% Expanded Uncertainty
False accept
y = x + e (observed value)
The probability of being in-tolerance is depends on more than just the measurement uncertainty.
If we move the tolerance limits out so that the entire population is within tolerance, the probability of being in-tolerance is 100% regardless of the measured value.
Applying a 95% expanded uncertainty guard band does not change that.
How, then, is the probability, given a measured value determined?
x (true value)

8. False Accept Risk, Given an Observed Value
tolerance limit
y = x + e (observed value)
False accept
Conditional risk: risk of false accept given a measured value.
Unconditional risk is the average risk for the population.
The risk increases as the measured value approaches the acceptance limit. The measurement uncertainty distribution is a close approximation to the conditional probability density function. (Offset in the mean, slight difference in the standard deviation, assuming Gaussian distributions.)
Need to create statement to transition out of this slide.
x (true value)

9. Risk Management (differing approaches)
Conditional Risk Management
False-accept risk management for a device given its observed value.
Unconditional Risk Management
Average false-accept risk management of a population. No statement is made regarding the false-accept risk of an individual device.
Restate the problem: How can we state pass/fail given these two different philosophies.

10. Pass/Fail Criteria Differing Risk Management Philosophies
Conditional versus Unconditional Risk Management
Varying Acceptance Limit Requirements
Guard Bands
Differing Outcome Labels
Pass/Fail/Indeterminate (ILAC-G8) versus Pass/Fail

11. A Common Approach to Outcome Labeling
Allow the acceptance limit to be set based on application specific criteria.
Define “Acceptance Limit” as simply the decision criteria for stating either Pass or Fail.
Annotate Pass/Fail labels to indicate “confidence level” interpretation.
Outcome labels: Pass, Pass1, Fail, Fail1

12. Zero Guard Band Fail zero guard band
Nominal
Lower acceptance limit and tolerance limit
Upper tolerance limit and acceptance limit
zero guard band
Pass
Pass1
Fail1
Fail
Measured value
95% expanded uncertainty

13. 2% False Accept Risk Guard Band
Nominal
Lower acceptance limit
Lower tolerance limit
Upper acceptance limit
Upper tolerance limit
2 % unconditional false accept risk guard band
Pass
Pass1
Fail1
Fail
Measured value
95% expanded uncertainty
Method 6.
This guard band allows meeting and Z540.3 simultaneously.

14. 95% Expanded Uncertainty Guard Band
Nominal
Lower acceptance limit
Lower tolerance limit
Upper acceptance limit
Upper tolerance limit
95 % expanded uncertainty guard band
Pass
Fail1
Fail
Measured value
95% expanded uncertainty
Method 5.

15. Conclusion
This method for expressing statements of Pass/Fail conformance provides key information for managing conditional and unconditional risk and meets global requirements.
ISO/IEC 17025
ANSI/NCSL Z540-1
ILAC-G8
Allows meeting when using the ILAC-G8 criteria and Z540.3 using a single method.
ANSI/NCSL Z540.3
EURAMET/cg-15/v.01
(previously EA 04/13)