1. COP pass/fail decisions
Norbert E. Ligterink

2. pass-fail decision schemes
number of tests
failures in PASS
failures in FAIL 3 - 4 1 …
COP pass/fail decisions
pass-fail decision schemes
sequential scheme based on:
producer and consumer risks
level of confidence (representativeness versus test burden)
Schemes can be adapted to match the general criteria
General criteria:
minimal number of tests:
all-pass with 3 or 5 tests
maximal number of tests:
for example: 11 or 17 tests
symmetry between producer and consumer risks:
11 tests: 5 or less faulty  pass, 6 or more faulty  fail
asymmetry: higher producer risk: e.g. 3 or less faulty  pass
 scheme can be adapted to meet criteria
example
“all-pass” with 3 tests means, e.g.:
less than 38% faulty with 95% confidence
less than 47% faulty with 90% confidence

3. COP pass/fail decisions
effect of the criteria
minimum number of tests leads to high confidence (easy to fulfil)
maximal number of tests is critical: (difficult to fulfil)
confidence increases slowly with the number of tests:
pass-with-40%-faulty with symmetric risk: (thus corresponding to fail-with-60% faulty)
pass at 11 tests with less than 6 faulty  75% confidence
pass at 17 tests with less than 9 faulty  80% confidence
asymmetry might be appropriate: (e.g. EPA)
e.g. pass 17 tests less than 4 faulty (higher producer risk)
Decision on min-max-symmetry determines scheme.
recommendation: minimal 3 tests, maximal 11 tests,
pass with “maximally-4-fails-in-11” asymmetry

4. background on pass-fail scheme statistics
COP pass/fail decisions
background on pass-fail scheme statistics
calculate the probability that a test is passed contrary to expectation based on the fraction of faulty (pfaulty)
 P(k failures out of n tests) = (n!/(k!(n-k!)) * pfaultyk *(1-pfaulty)n-k
(similar for fail decision with a fraction of non-faulty)
pass decision on at most K failures:
P(k=0) + P(k=1) + … + P(k=K) < 1 – confidence level
confidence level
pfaulty
higher confidence levels for lower faulty fractions