1. Preventing OOS Deficiencies
Lynn Torbeck
Thank you
And thanks to the PDA for making this web seminar possible today
4/14/2017 9:14 AM
Preventing OOS Deficiencies

2. Preventing OOS Deficiencies
List of Topics
Briefly review:
Barr Case
FDA OOS Guidance
Able Laboratories Story
PDA Scientific Advisory Board Committees
Troublesome fundamentals
Unresolved issues
Preventing OOS deficiencies
Final recommendations
Here is a list of topics that we will discuss in today’s session.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

3. Preventing OOS Deficiencies
Barr Case
Audited in 1989, 1991 and 1992.
Refused to accept a consent decree.
FDA was forced to go to court.
Civil action taken June 1992.
Decision in favor of the FDA on February 4, 1993.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

4. Barr and Statistical Issues
Initial investigations
Full investigations
Testing
Retesting
Averaging
Outliers techniques
There many issues in the Barr case, but here are some of the issues we will cover relative to OOS.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

5. Preventing OOS Deficiencies
This is a flowchart of the logic for doing both initial and full investigation.
I know this is a little small, so I will discuss each item.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

6. Preventing OOS Deficiencies
Barr: Lessons Learned
FDA takes OOS issues very seriously.
OOS SOP’s, laboratory logs and documented investigations will be part of any Quality System review.
Companies are still getting Form 483 observations for not having an adequate SOP or for not following the SOP.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

7. Preventing OOS Deficiencies
Barr: OOS Prevention
Analysts, supervisors and managers should read and discuss the Barr case and understand the OOS issues in context.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

8. Preventing OOS Deficiencies
FDA Guidance
“Investigating Out of Specification (OOS) Test Results for Pharmaceutical Production.”
Issued as a draft in September 1998.
Still in draft as of today.
FDA has sent it to the attorneys.
Final version could be out this year.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

9. Preventing OOS Deficiencies
Draft: OOS Prevention
All laboratory personnel, analysts, supervisors and managers should read, study and discuss in-depth, sentence by sentence if necessary, the draft OOS guidance.
Then do it again when the final guidance is released.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

10. Preventing OOS Deficiencies
Able Labs – Cranbury, NJ
Massive number of OOS errors
Recall of all 46 products
3,184 lots recalled
Five ANDA’s withdrawn
Hundreds of staff laid off
Sold to Sun Pharm in December 2005
4/14/2017 9:14 AM
Preventing OOS Deficiencies

11. Able Labs: Lessons Learned
It is still possible to have wide spread misunderstanding of the Barr case, the OOS guidance and OOS SOPs.
Apparently the analysts felt they could not give an “incorrect result.”
Management needs to instill and cultivate a “GMP Culture” in the analytical laboratory.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

12. Able Labs: OOS Prevention
Review the Able Labs web site.
Discuss the Able Labs story with laboratory analysts, supervisors and managers.
Discuss what a “GMP Culture” means in the analytical laboratory and how to develop and reward it.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

13. Preventing OOS Deficiencies
PDA OOS Committees
Chemical OOS:
Lynn Torbeck, Chair
Eight members
Draft technical report reviewed by the FDA
Planning a PDA/FDA conference
Microbial Data Deviations:
Jeanne Moldenhauer, Chair
Draft in revision
4/14/2017 9:14 AM
Preventing OOS Deficiencies

14. Troublesome Fundamentals:
Outliers
Reportable Values
Averaging
Testing into compliance
Full consideration
4/14/2017 9:14 AM
Preventing OOS Deficiencies

15. Preventing OOS Deficiencies
Outliers - Defined
Extreme values vs outliers:
Statisticians draw the distinction between an extreme value and an outlier.
An extreme value while large or small can be considered to be part of the data set.
An outlier is so far removed from the data set that we believe that is not part of the data set.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

16. Preventing OOS Deficiencies
Outliers – Judge Wolin
"The USP expressly allows firms to apply this test (outlier) to biological and antibiotic assays, ..., but is silent on its use with chemical tests.”
"In the Court's view the silence of the USP with respect to chemical testing and outliers is prohibitory."
In the Barr case Judge Wolin drew an interesting distinction.
Not everyone agrees with that position, but at the same time it has not been changed.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

17. Outliers - Investigation
"In chemical procedures, where method accuracy variation is small, an outlier test may be appropriate as part of an OOS investigation, provided the sample and test procedure assumes homogeneity ... as in the composite strength assays. Our current thinking is that outlier tests are never appropriate where the purpose of the sample is to measure uniformity" Paul Vogel, September 10, 1993.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

18. Preventing OOS Deficiencies
Outliers - Tests
Dixon's criteria, the test in USP<111>, is general in nature and not specific to biological issues. It can be used anywhere the statistical assumptions can be met.
In general, statisticians agree with the philosophy that outlier tests should be used infrequently and with great caution.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

19. Outliers - Recommendations
Don't use any outlier rejection test for rejection of chemical test results. But it can be used as supporting information in an OOS investigation to consider retesting.
Keep all data, especially suspect data, for future review. Unusual data when seen in context and with other historical data often is not unusual at all, but in fact forms a known and well-behaved statistical distribution.
What looks like a normal distribution with outliers, may turn out to be a log-normal distribution.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

20. Preventing OOS Deficiencies
Reportable Values
“Reportable Values for Out of Specification Test Results”
Lynn Torbeck
Pharmaceutical Technology
Vol. 23, No. 2, February 1999
Special Supplement
4/14/2017 9:14 AM
Preventing OOS Deficiencies

21. Preventing OOS Deficiencies
FDA R.V. Definition
“It should be noted that a test might consist of replicates to arrive at a result. For instance, an HPLC assay result may be determined by averaging the peak responses from a number of consecutive, replicate injections from the same preparation. The assay result would be calculated using the peak response average.”
In that paper I pointed out that the FDA defined a reportable value
4/14/2017 9:14 AM
Preventing OOS Deficiencies

22. Preventing OOS Deficiencies
FDA R.V. Definition
“This determination is considered one test and one result.”
4/14/2017 9:14 AM
Preventing OOS Deficiencies

23. Implications of FDA Definition
A reportable value is the end result of the complete measurement method as documented.
It is the value compared to the specifications.
It is the value used for official reports.
It is usually the value used for statistical analysis.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

24. Preventing OOS Deficiencies
Figure 1
I came to this subject at the PDA annual meeting in the fall of 1998 just after the guidance was issued.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

25. Preventing OOS Deficiencies
Figure 2
4/14/2017 9:14 AM
Preventing OOS Deficiencies

26. Preventing OOS Deficiencies
Figure 3
4/14/2017 9:14 AM
Preventing OOS Deficiencies

27. Preventing OOS Deficiencies
Interpretation
The individual determinations do not have to meet the specification.
Individual determinations are not reported out of the lab.
However the variability of the determinations is a system suitability issue.
Set a limit on the standard deviation or %RSD.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

28. Preventing OOS Deficiencies
R.V.: OOS Prevention
Record in writing the operational definition of the Reportable Value for each test method in the method documentation, any protocols and any reports.
Add “Only this reportable value can be compared to the specification criteria.”
For example, “The RV for this test method is the average of single injections from three preparations of the sample delivered to the laboratory.”
4/14/2017 9:14 AM
Preventing OOS Deficiencies

29. Preventing OOS Deficiencies
Averaging
Specifically, the arithmetic mean; the sum of all of the numbers divided by the count of the numbers. 
More generally, it is a value that represents the central point of a data set. (In this sense, it can include the arithmetic average, the median, the mode, the geometric mean or the harmonic mean.)
We can discuss this in the context of reportable values.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

30. Preventing OOS Deficiencies
Averaging
"... as a general rule, firms should avoid this practice, because averages hide the variability among individual test results.“
"[Averaging] is particularly troubling if testing generates both out‑of‑specification and passing individual results which when averaged are within specification. 
"Here, relying on the average figure without examining and explaining the individual out‑of‑specification results is highly misleading and unacceptable."
4/14/2017 9:14 AM
Preventing OOS Deficiencies

31. Preventing OOS Deficiencies
Averaging
"Averaging the results of tests intended to measure the uniformity of the test article is not current good manufacturing practice ... because it may hide the variability of the sample the test procedure is intended to detect. For this reason, all individual test results must be reported and evaluated on an independent basis" Paul Vogel, September 10, 1993.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

32. Averaging: OOS Prevention
Do not average out of specification reportable values within specification reportable values to get an in specification result.
Do not average reportable values for QA to make a decision. QA must see all individual reportable values, OOS and retests.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

33. Testing Into Compliance
Torbeck, L., “Preventing the Practice of Testing into Compliance”, Pharmaceutical Technology, Oct 2002.
Testing into compliance is the practice of ignoring valid information that should be used to make decisions.
Such a practice is at best not scientific and at worst is fraudulent, illegal, and immoral.
Such practices if found must be stopped.
Hot button
4/14/2017 9:14 AM
Preventing OOS Deficiencies

34. Testing Into Compliance
Averaging OOS results with in specification results to get an in specification result.
Physically averaging powers, granulations and liquids to get in specifications results.
If not part of the validated process.
Discarding data or not recording data until is known to be in specification.
Missing samples and rejected cans.
Overwriting HPLC chromatograms.
Examples:
If time do the good cans and bad cans
4/14/2017 9:14 AM
Preventing OOS Deficiencies

35. Not Testing Into Compliance
Large initial sample sizes are acceptable if all data generated is reported.
Large number of retests are acceptable if all data generated is reported.
Failing system suitability is not an OOS.
Out of limits for an in-process adjustment is not an OOS.
Because there is misunderstanding about testing into compliance, some people thing that just taking large samples is a problem.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

36. Compliance: OOS Prevention
Train all laboratory personnel, analysts, supervisors and managers to be able to identify specific situations of testing into compliance.
Train to be able to defend situations that are not testing into compliance during an audit.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

37. Preventing OOS Deficiencies
Full Consideration
“For inconclusive investigations …. The OOS result should be retained in the record and given full consideration in the batch or lot disposition decision.
This statement has caused some discussion as it is considered to be vague and undefined. It can, I think, be defined in a simple way.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

38. Preventing OOS Deficiencies
Full Consideration
First, all QA decisions are made with the Reportable Values, both OOS and retests.
Second, QA looks at the magnitude of the retest values compared to the specifications.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

39. Preventing OOS Deficiencies
Full Consideration
If the retest values are close to the target, the lot can be released.
If the retest values are close to the limit that the OOS exceeded, technically the lot can be released, but QA should consider further investigation to determine why the retests are not at target.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

40. Consideration: OOS Prevention
QA should detail and document the logic and rational for decisions based on retesting results after a OOS result is found.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

41. Preventing OOS Deficiencies
Unresolved Issues
Specification Limits for OOS?
What size the retest sample?
Second analyst?
Statistical treatment of data?
4/14/2017 9:14 AM
Preventing OOS Deficiencies

42. Specification Limits for OOS?
Regulatory Limits
Release: accept/reject
Action limits, Cpk=1.33
Alert, Cpk=1.0
Warning limits
Trend
Validation limits
Most companies have several levels of specifications
Regulatory, accept/reject, action and alert are the most common.
These can be seen in relation to each other in this figure.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

43. Preventing OOS Deficiencies
Specification Limits
Here the innermost limits are the alert limits.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

44. Specification: OOS Prevention
Define in writing the levels of specification criteria.
Justify in writing which specifications are considered applicable to OOS and why or why not.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

45. What Size the Retest Sample?
“… a matter of scientific judgment,”
“… retesting cannot continue ad infinitum.”
“Such a conclusion cannot be based on on 3 of 4 or 5 of 6 passing results, but possibly 7 of 8.”
“… will vary on a case by case basis … “
“… an inflexible retesting rule … is inappropriate.”
4/14/2017 9:14 AM
Preventing OOS Deficiencies

46. What Size the Retest Sample?
“The number of retests … should be specified in advance …”
“The number of tests should not be adjusted ‘on-the-fly’, as results are being generated.”
“… a firm’s predetermined testing procedure should contain a point at which testing ends and the product is evaluated.”
4/14/2017 9:14 AM
Preventing OOS Deficiencies

47. What Size the Retest Sample?
This is an unresolved issue and the statisticians are still publishing journal articles and discussing it.
Barr case n=7.
Could be too much or not enough.
Currently n= 3 to n=9.
PDA OOS committee will recommend.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

48. Preventing OOS Deficiencies
Retest References
Hofer, J., Considerations when determining routine sample size for a retest procedure, Pharmaceutical Technology, Nov
Anderson, S., An alternative to the ESD approach, Pharmaceutical Technology, May 2004.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

49. Retest: OOS Prevention
Define in writing the sample size for retests or define the procedure to be used to determine the sample size.
Provide scientific justification.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

50. Preventing OOS Deficiencies
Second Analyst
Guidance suggests a second analyst.
Issues:
Added complication and variation
May not have a second analyst
May not find the root cause
Second analyst may not be as proficient
Recommend that the manager decide and justify decision in writing.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

51. Statistical Treatment of Data
“Statistical treatments of data should not be used to invalidate a discrete chemical test result.”
“ … a statistical analysis may be valuable as one assessment of the probability of the OOS result …
Another way to say outlier rejection.
It would be possible to build statistical models to identify values as being outliers.
Control charts, ANOVA, regression, etc.
The point being that the statistical tools are there to help gather data and not as decision rules.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

52. Preventing OOS Deficiencies:
Setting specification criteria
Statistical Thinking
Sources of variation
Common cause vs. special cause
Variation reduction
Training
Education
4/14/2017 9:14 AM
Preventing OOS Deficiencies

53. Setting Specification Criteria:
Two sides to the OOS issue.
Incorrect limits are the major source of OOS.
Many specifications were set early in the development process and may not be appropriate for the current process.
Many specification were set using wishful thinking or incorrect approach.
Two sides are the RV and the Spec.
If the spec are set incorrectly, you never find a root cause.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

54. Setting Specification Criteria:
Use historical data
Use distribution analysis
Normal, log-normal, exponential
Don’t use X bar  3S
Use Statistical Tolerance Intervals
X bar  K S for the alert limits
where K is based on confidence and percent of future values
Contact your local statistician
For example, We could say “I am 99% confident that 99% of future value will lie within these limits.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

55. Setting Specification Criteria
For action limits, permit the average to vary and widen the Tolerance Limits
For accept/reject limits, add a further allowance for stability.
Consider the clinical results as part of the justification for limits.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

56. Preventing OOS Deficiencies
Statistical Thinking
All work occurs in a system of interconnected processes.
All processes have variability.
Process understanding and variability reduction is the key to success.
Variation is the enemy.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

57. Preventing OOS Deficiencies
Sources of Variation:
Common cause variation:
People
Materials
Methods
Measurement
Machines
Environment
Special cause variation:
One single factor changed
4/14/2017 9:14 AM
Preventing OOS Deficiencies

58. Common vs. Special Causes
A plot of the data with natural limits illustrates common cause variation.
A value that is larger than would be expected by chance alone is assumed to be due to a special cause. Use CAPA to find it.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

59. Variability Reduction:
Display boards
Operational definitions
Work to target, Target ( Low, High )
Flexible consistency
Hold constant
Mistake proofing
High tech equipment
Add stories
4/14/2017 9:14 AM
Preventing OOS Deficiencies

60. Preventing OOS Deficiencies
Training
Training is for a specific task or SOP.
The goal is consistency.
Freelancing causes problems.
Little background is provided.
An in-depth understanding is not needed to be in compliance if the SOP is followed.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

61. Preventing OOS Deficiencies
Education
Someone needs to:
Learn and understand the basic philosophy and principles.
Know the background as it relates to the topic.
Understand the material well enough to be able to make difficult decisions with confidence and be able to defend them.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

62. Need for Understanding
Why was Able Labs out of compliance?
Defend Reportable Values.
Defend specifications applicable to OOS
Defend not testing into compliance.
Defend retest sample size.
Why variability reduction is needed.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

63. Final Recommendations
Read and understand the Barr Case.
Read and study in-depth the OOS Guidance. Once is not enough.
Audit the company SOP against the Guidance line by line.
Have an active program to reduce OOS results.
Keep management informed.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

64. Preventing OOS Deficiencies
Thank You
That ends my presentation.
We are now ready for questions and answers.
4/14/2017 9:14 AM
Preventing OOS Deficiencies

65. Preventing OOS Deficiencies
References
USA vs. Barr Laboratories, Inc. Civil Action No , US District Court for the district of New Jersey, February 4, 1993.
FDA, CDER, “Guidance for Industry, Investigating Out of Specification (OOS) Test Results for Pharmaceutical Production,” September 1998.
Torbeck, L., “Reportable Values for Out-of-Specification Test Results,” Pharmaceutical Technology, February 1999.
Torbeck, L., “Preventing the Practice of Testing into compliance,” Pharmaceutical Technology, October 2002.
Hahn, G and Meeker, W., Statistical Intervals, John Wiley & Sons, 1991.
Torbeck, L., “Statistical Thinking,” Pharmaceutical Technology, July 2001.
4/14/2017 9:14 AM
Preventing OOS Deficiencies