STATISTICAL BACKGROUND FOR CBER PROPOSALS FOR ACCEPTABLE PROCESS CONTROL PLANS John Scott, Ph.D. Division of Biostatistics FDA / CBER / OBE October 19, 2012

Outline  Statistical Quality Control overview  Sampling for quality control  Single-stage sampling plans  Binomial plans  Hypergeometric plans  Double-stage sampling plans  Binomial plans  Hypergeometric plans  Further possibilities 2

Statistical Quality Control Overview 3

What is quality control?  Quality = “fitness for purpose”  Many aspects to assessing quality, e.g.  Performance, reliability, durability, serviceability, aesthetics, features, perceived quality, conformance to standards  Much of the quality control literature centers on reduction of variability  We’re focused on assessing conformance  Non-conformance is the end result of uncontrolled variability 4

Methods of statistical quality control  Statistical process control methods  Describe and explain variability in an ongoing process  Methods focus on graphical displays – simple plots, Pareto charts, control charts  Experimental design methods  Systematically assess sensitivity of outputs to changes in input  Acceptance sampling methods  Inspect and test final product for conformance 5

Some SPC graphs Stem and leaf plotBox plotPareto chart 6

Control charts  Control charts map samples of a product characteristic over time to identify when a process might be out of control  Features:  A center line corresponding to the average “in control” value  Upper and lower control limits  Out of control values or unusual runs may be signals for investigation or action 7

Control chart example 8

Sampling for QC 9 Population Sample

Conformance to standards  The plateletpheresis and leukoreduction Guidances each recommend:  Standards for QC testing of individual units  A maximum acceptable proportion of non-conforming units  A minimum confidence level  Only process failures are counted as non-conforming  Non-process failures (failures due to uncontrollable parameters) are replaced in QC testing 10

Leukoreduction standards 11

Plateletpheresis standards 12

Maximum process failure rates  5% for:  Residual WBC content  Content recovery / retention  pH  25% for:  Platelet yield 13

Minimum confidence levels  In each case, the Guidances recommend establishing a process failure rate below the maximum “with 95% confidence”  That means: if the true process failure rate for a given month is at the maximum allowable level, there’s:  A 95% chance that QC testing will detect a problem, or  A 5% chance that the problem will go erroneously undetected  95% confidence in a 95% success rate is called “95%/95% acceptance”  95% confidence in a 75% success rate is “95%/75% acceptance” 14

How to get to 95% confidence  There’s an “easy” way to have 100% confidence: test every unit. E.g.:  Suppose you produce 200 units of RBC, LR in a given month  You test residual WBC content on every unit  All but 8 units meet WBC < 5.0 x 10 6  The failure rate is 8/200 = 4% with 100% confidence  When testing every unit is impractical (i.e. usually), you can use statistical sampling to get to 95% confidence 15

The logic of statistical sampling  The population is what you want to draw a conclusion about  E.g. this month’s RBC units  Take a random sample, measure failure rate in the sample, use probability models to make inference about population rate  Yields an estimate of population rate  Estimate has some uncertainty because sample is incomplete Population Sam- ple 16

Acceptance sampling  Acceptance sampling consists of taking a sample from a lot and making a decision to accept or reject the lot based on the sample  Not the same as SPC for blood establishments  A month’s worth of units is not a “lot”  A month’s worth of units will not be discarded on failed QC  Common methods of acceptance sampling do correspond to FDA recommendations 17

Acceptance sampling translations  In order to use acceptance sampling terminology, we need a couple translations  “Accept” =  Pass QC testing for the month  No further action required  “Reject” =  Fail QC testing for the month  Launch a failure investigation 18

Single-stage sampling plans 19

Single-stage sampling plans  In a given month, N units will be produced  In a single-stage sampling plan:  A sample of n units is tested  If more than c process failures, reject; otherwise accept  E.g. n = 60, c = 0 means:  We test a sample of 60 units  If no process failures, accept  If at least 1 process failure, reject, launch a failure investigation  n and c need to be chosen to provide recommended confidence 20

Binomial single-stage sampling  The Guidances recommend that n and c be chosen such that the probability of accepting is at most 5% when the true failure rate is at the maximum allowed  We need a probability model to calculate the probability of accepting  For large N we usually use the binomial distribution  Called “Type B sampling” in SPC literature  The binomial distribution assumes a random sample is taken with replacement from an infinite population 21

*The binomial distribution  Assume:  The true failure rate is p  The sample size is n  The acceptance number is c  Then, under binomial sampling, the probability of accepting is given by:  For example, p =.05, n = 60, c = 0 gives a probability of accepting of 22

Operating characteristic curves  For any sampling plan, we can ask: what’s the probability of accepting at any given true failure rate?  Recall that the Guidances recommend, e.g., 5% probability of accepting at a true residual WBC failure rate of 5%  We also care about the probability of accepting at a “good” failure rate  An operating characteristic (OC) curve plots the probability of accepting against the true failure rate 23

OC curve examples 24

Practical binomial single-sampling  Very few binomial single-stage sampling plans are useful in practice  Once you choose c, the smallest allowable n is the best choice  For 95%/95%: 1. c = 0, n = 59* 2. c = 1, n = c = 2, n = 124 *note that the Guidances mention c = 0, n = 60; establishments often prefer the round number, but c = 0, n = 59 is also acceptable 25

Binomial sampling key points  Should be used when N is large  Also appropriate for process validation  n and c should be chosen to meet 95/95 or 95/75:  Larger values of c and n mean lower false positive rates 26 cn (95/95)n (95/75)

Hypergeometric single-stage sampling  The binomial distribution assumes an infinite population  You don’t produce an infinite number of units, but this works well enough for large N  If the population is small, we can use the hypergeometric distribution instead  Called “Type A sampling” in SPC literature  The hypergeometric distribution assumes a random sample is taken without replacement from a finite population of size N 27

*The hypergeometric distribution  Assume:  The population size is N  The population number of successes is m (m ≥ (1-p)N)  The sample size is n  The acceptance number is c  Under hypergeometric sampling, the probability of accepting is given by: 28

Hypergeometric OC curve examples 29

Binomial vs. hypergeometric plans  Hypergeometric plans have logistical difficulties:  You need to know N (or put upper bound on N)  QC process may change from month to month or component to component  Binomial plans require larger samples  As N gets large, difference between binomial and hypergeometric approaches narrows  Almost no difference when N is at least 10 times bigger than n 30

Hypergeometric key points  Useful when N is small  Sampling plan depends on knowing N  You may need to plan around an upper bound on N  Can’t be used for process validation  For each N, you can choose n and c to meet 95/95 or 95/75. E.g. for N = 100: 31 Cn (95/95)n (95/75)

Single-stage sampling issues  The sampling plan needs to be prespecified  Not acceptable to plan on c=1, n=93, then get to 59 with no process failures and accept without further testing  You need to sample in such a way that you’ll have enough tests to meet QC acceptance rule  With a c=1, n=93 sampling plan, if you have 1 process failure in 80 tests in a month, 95%/95% hasn’t been met  For small N, hypergeometric plans may involve testing almost 100% of units  You may need to test consecutively from start of month  Potentially problematic if something changes mid-month 32

Double-stage sampling plans 33

Double-stage sampling plans  In a double-stage sampling plan:  A sample of n 1 units is tested  If c or fewer process failures are observed, accept  If c+2 or more process failures are observed, reject  If c+1 process failures are observed: A second sample of n 2 units is tested If no more process failures are observed, accept If one or more process failures are observed, reject  c, n 1 & n 2 are chosen to meet recommended criteria (e.g. 95%/95%, 95%/75%) 34

Binomial double-stage plans  For large N, we calculate probability of acceptance using the binomial distribution  Some acceptable double-stage sampling plans: 35 C95/9595/75 N1N1 n2n2 n1n1 n2n

Double vs. single-stage sampling 36

Flexibility in double-stage plans  There are more choices with double-stage plans  All of these meet 95%/95%: cn1n1 n2n2 Pr(accept) if p = 1% Avg. total n if p = 1% Max n % % % %

Hypergeometric double-sampling  As with single-stage sampling, binomial double-stage sampling is pretty efficient if N is large  For small N, hypergeometric double-stage sampling may allow smaller samples  You need to know N (or be able to put an upper bound on it)  The calculations are a little complicated; Appendix A of the leukoreduction guidance provides a table 38

Hypergeometric double-stage sampling examples 39

Double-stage sampling issues  As with single-stage sampling:  Pre-specification required  Need to ensure enough units are tested  Note that many hypergeometric plans have a second stage of 100% (“ALL”) or almost 100%  You can’t get to 100% if you skipped any  For N=50, c=0, n 1 =31, n 2 =18, you may be done with testing two weeks into the month Will you catch issues late in the month? Supplemental testing may be advisable 40

Double-stage sampling key points  Adopting a double-stage sampling plan:  Gives you a chance to “rescue” a process in the event of a failed test  Reduces risk of false positives  Binomial plans are appropriate for large N  Hypergeometric plans yield smaller sample sizes for small N  Can’t be used for process validation  Plans should be pre-specified 41

Other Possibilities 42

Multiple-stage sampling  Double-stage sampling can be generalized:  The second stage can be designed to allow for 1 or more process failures  Third, fourth, or … stages can be added  None of these techniques are likely to lead to practical sample sizes in the blood establishment setting 43

Chain sampling  In chain sampling, a sample of size n would be taken each month  If 0 process failures, accept  If 2 process failures, reject  If 1 process failure:  Accept if there have been 0 process failures in the past i months  Otherwise reject  Hasn’t been proposed, to my knowledge 44

Scan statistics  Scan statistics can be used to continuously monitor an ongoing process to look for event clusters  How it works:  N tests over a year  Look at every possible sequence of m consecutive tests (a “window”)  If there are more than k failures in a window, the process is out of control  N, m and k chosen to achieve probability of declaring: In control at a “good” failure rate, p 1 (e.g. 1%) Out of control at a “bad” failure rate, p 2 (e.g. 5%) 45

Scan statistic pros and cons  A natural way to look at blood product QC  Months are arbitrary (although c.f. CFR)  Avoids issues related to non-random sampling  Mathematically & logistically complex  No free lunch!  Added flexibility of the scan statistic comes at a price  If you want 95%/95% for any scan window, large N (e.g. N > 1000) required, or false positives high  Compare to 60/month binomial N =

Thanks! 47