1. Copyrights © 2006 Silliker, Inc. All Rights Reserved Challenges In Validating Analytical Methods in an Independent Lab

2. Copyrights © 2006 Silliker, Inc. All Rights Reserved 2 Overview The purpose of this presentation is to review the process and highlight the complexity of method validation in a third party lab.

3. Copyrights © 2006 Silliker, Inc. All Rights Reserved Lab Function

4. Copyrights © 2006 Silliker, Inc. All Rights Reserved 4 What is the laboratory’s function? Receive samples Perform tests Deliver results

5. Copyrights © 2006 Silliker, Inc. All Rights Reserved 5 What does a laboratory need to do? Consistently deliver data that is  Accurate  Reliable  Valuable

6. Copyrights © 2006 Silliker, Inc. All Rights Reserved 6 Lab Assumptions Methods validated  For specific matrix Quality system in place for analysis  Analysts  Facility  Analysis tool Sampling plans are appropriate

7. Copyrights © 2006 Silliker, Inc. All Rights Reserved The Four BIG Questions: What? Why? How? Where?

8. Copyrights © 2006 Silliker, Inc. All Rights Reserved 8 What is our product and purpose? A laboratory produces data Data is used to make decisions

9. Copyrights © 2006 Silliker, Inc. All Rights Reserved 9 Microbiological Data may be Used to Assess: The safety of food Verification/validation procedures in HACCP Adherence to GMP/GHP The utility (suitability) of a food or ingredient for a particular purpose The keeping quality (shelf-life) of certain perishable foods Acceptability of a food or ingredient from a source for which there is not confidence in the process

10. Copyrights © 2006 Silliker, Inc. All Rights Reserved Validation Challenges

11. Copyrights © 2006 Silliker, Inc. All Rights Reserved 11 Changing Technology Inherent difficulty for labs to take advantage of technical improvements Labs should have the ability to bring technical innovation to the user Keeping up with technology leads to increased costs

12. Copyrights © 2006 Silliker, Inc. All Rights Reserved 12 Regulatory Issues In the event of a regulatory action, validated testing is a virtual necessity In the event of a legal dispute, validated testing is also a virtual necessity Juries trust validated tests more than non validated

13. Copyrights © 2006 Silliker, Inc. All Rights Reserved 13 Method Comparison / Validation Issues Most properties can be measured When 2 or more alternative methods exist for measuring the same property, how do you compare ? Do they really measure the same thing ?

14. Copyrights © 2006 Silliker, Inc. All Rights Reserved 14 Method Comparison / Validation Issues Philosophically, different methods can’t measure exactly the same thing No measuring technique responds to only one single property Relationship of methods could very well depend on the material being measured

15. Copyrights © 2006 Silliker, Inc. All Rights Reserved 15 What is considered Functional relationship between alternative methods Technical and Economic Merit  Can be >$ 200,000 US  Is the change worth the cost.

16. Copyrights © 2006 Silliker, Inc. All Rights Reserved 16 Biggest Issues Huge diversity of sample types Most commercial methods are validated against a limited number of analytes Sample prep isn’t considered Sample compositing isn’t considered in most validations

17. Copyrights © 2006 Silliker, Inc. All Rights Reserved Validation Process

18. Copyrights © 2006 Silliker, Inc. All Rights Reserved 18 Validation Valid “ well grounded or justifiable ; logically correct “

19. Copyrights © 2006 Silliker, Inc. All Rights Reserved 19 Validation Validate “ to support or corroborate on a sound or authoritative basis”

20. Copyrights © 2006 Silliker, Inc. All Rights Reserved 20 Validation “ an act, process to determine the degree of validity of a measuring device”

21. Copyrights © 2006 Silliker, Inc. All Rights Reserved 21 Validation Process Overview Measurement Evaluation Verification Summary

22. Copyrights © 2006 Silliker, Inc. All Rights Reserved 22 “An element of chance enters into every measurement ; hence every set of measurements is inherently a sample of certain more or less unknown conditions. Even in those few instances where we believe that the objective reality being measured is a constant, the measurements of this constant are influenced by chance or unknown causes.” W.A. Shewart

23. Copyrights © 2006 Silliker, Inc. All Rights Reserved 23 No two things are alike, but even if they were, we would still get different values when we measured them. W.A. Shewart

24. Copyrights © 2006 Silliker, Inc. All Rights Reserved 24 Overview Microbiological analysis will continue to be a cornerstone used to determine the safety and quality of foods in domestic and international trade Microbiological data are important to determine compliance with Food Safety Objectives, microbiological criteria, and for HACCP validation and verification

25. Copyrights © 2006 Silliker, Inc. All Rights Reserved 25 Overview (cont.) Microbiological data used to determine acceptability of products in domestic and international trade must be reliable and consistent among trading partners Both regulators and industry need to maximize the capacity and credibility of laboratory testing for both official and routine purposes

26. Copyrights © 2006 Silliker, Inc. All Rights Reserved 26 Process Choices Manual  Human – majority of tests Semi Automated  Human / Machine - growing percentage Automated  On its way

27. Copyrights © 2006 Silliker, Inc. All Rights Reserved Measurement

28. Copyrights © 2006 Silliker, Inc. All Rights Reserved 28 What is Measurement 4 scales  Nominal, Ordinal, Interval, Ratio Relationship to some property  Direct or indirect Production process  Sampling through to Decision making Performance characteristics  Rugged, Practical, Specific, Reliable

29. Copyrights © 2006 Silliker, Inc. All Rights Reserved 29 Measurement Considerations Measurement unit reflects variation Consistent over time Unbiased Characterize product relative to spec limits Reflect product that has not been measured

30. Copyrights © 2006 Silliker, Inc. All Rights Reserved 30 Measurement Considerations Usefulness in process control Detects differences Technique comparison Product information from measurement

31. Copyrights © 2006 Silliker, Inc. All Rights Reserved Microbiological Testing Applications Water testing Environmental pathogen programs Incoming ingredient testing Finished product analysis Pathogenic organisms Spoilage organisms Finished product challenge studies Process validation studies

32. Copyrights © 2006 Silliker, Inc. All Rights Reserved 32 Evaluation Process Design the study  Appropriate to deliver needed info Choose matrix  Sample to test Choose methods  New vs. old Choose measurement instruments  Humans included Choose reference material  If possible Perform statistical analysis

33. Copyrights © 2006 Silliker, Inc. All Rights Reserved 33 Statistical performance Standard deviation Repeatability Reproducibility Operator bias Operator error Test bias Test error

34. Copyrights © 2006 Silliker, Inc. All Rights Reserved 34 Method verification and proficiency testing are essential components of a laboratory’s quality system and are necessary to determine Uncertainty of a microbiological data result

35. Copyrights © 2006 Silliker, Inc. All Rights Reserved 35 Method Validation - Reliability 1. Reproducibility – between lab precision. 2. Repeatability – within-lab precision. 3. Systematic error or bias – deviation from the ‘true’ value. 4. Specificity – ability to measure what is intended to be measured. 5. Limit of reliable measurement – smallest increment that can be measured with confidence. 6. Uncertainty in result (AOAC)

36. Copyrights © 2006 Silliker, Inc. All Rights Reserved Validation Types

37. Copyrights © 2006 Silliker, Inc. All Rights Reserved 37 Validations Overview Single Lab Intralaboratory Interlaboratory

38. Copyrights © 2006 Silliker, Inc. All Rights Reserved 38 Single Lab One lab - one matrix – one analyte Matrix - analyte specific method Extreme validity Difficult reproducibility “In –House “ methods

39. Copyrights © 2006 Silliker, Inc. All Rights Reserved 39 Intralaboratory Within a lab Somewhat like single validation Typical statistical measures are used to verify performance Normally cover multiple matrices for one analyte

40. Copyrights © 2006 Silliker, Inc. All Rights Reserved 40 Interlaboratory Throughout larger lab system Multiple sites Complex measure of ruggedness Both within and between lab variation measured Very Expensive

41. Copyrights © 2006 Silliker, Inc. All Rights Reserved Process

42. Copyrights © 2006 Silliker, Inc. All Rights Reserved 42 Testing Key step in process Success is dependent on previous steps Requires in depth planning Requires stringent quality systems

43. Copyrights © 2006 Silliker, Inc. All Rights Reserved 43 Testing Considerations Product knowledge Previous use Cost and value Analysis performance

44. Copyrights © 2006 Silliker, Inc. All Rights Reserved 44 Testing Method Selection  Analyte  Matrix  Time  Money Method Execution  Receipt  Prep  Test  Result

45. Copyrights © 2006 Silliker, Inc. All Rights Reserved 45 Types of tests Quantitative  Microbiological  Chemical Qualitative  Microbiological  Chemical

46. Copyrights © 2006 Silliker, Inc. All Rights Reserved 46 Validation Process - Quantitative Determines equivalence of methods for an analyte based on a numerical scale Determines proper testing conditions to achieve accurate results Determines appropriate field of use

47. Copyrights © 2006 Silliker, Inc. All Rights Reserved 47 Validation Process - Qualitative Determines equivalence of methods for an analyte based on a yes or no scale Determines proper testing conditions to achieve accurate results Determines appropriate field of use

48. Copyrights © 2006 Silliker, Inc. All Rights Reserved 48 Matrix Considerations What is the matrix being tested? What information do we need ?  Qualitative  Quantitative Are there matrix effects on the test?

49. Copyrights © 2006 Silliker, Inc. All Rights Reserved 49 Analyte Microbiological Chemical Physical

50. Copyrights © 2006 Silliker, Inc. All Rights Reserved Validation Examples

51. Copyrights © 2006 Silliker, Inc. All Rights Reserved 51 Study Design Key element of method comparison Qualitative or quantitative Choice of analyte Choice of samples size Inoculation levels

52. Copyrights © 2006 Silliker, Inc. All Rights Reserved 52 Example – qualitative Comparison of 2 Salmonella methods Gold standard vs. PCR PCR test uses proprietary enrichment media Question to answer “Is there a significant difference in the methods abilities to detect salmonella in X matrix ?”

53. Copyrights © 2006 Silliker, Inc. All Rights Reserved 53 What do we do? Determine appropriate test protocol  Diluents  Time  Temperature Determine matrices to test  Sample effects Determine inoculation levels to provide partial positive results  Strains and levels Determine if a limit of detection study is needed

54. Copyrights © 2006 Silliker, Inc. All Rights Reserved 54 Analysis Run each according to desired protocols Verify quality systems around the tests Statistical analysis  Chi square analysis for unpaired samples  Due to separate enrichment broths

55. Copyrights © 2006 Silliker, Inc. All Rights Reserved 55 Example - quantitative 2 methods to measure Aerobic plate count Gold standard vs. dry rehydrateable plate Question to answer “Is there a significant difference in the methods abilities to enumerate organisms in X matrix”

56. Copyrights © 2006 Silliker, Inc. All Rights Reserved 56 What do we do ? Determine appropriate test protocol  Prep  Time  Temperature  Enumeration procedure Determine matrices to test  Sample effects Determine inoculation levels to give a response over a large range  5 to 6 log range Determine if a limit of detection study is needed

57. Copyrights © 2006 Silliker, Inc. All Rights Reserved 57 Analysis Run each according to desired protocols Verify quality systems around the tests Statistical analysis  T test to look at the mean log difference in counts over a given range  Samples paired

58. Copyrights © 2006 Silliker, Inc. All Rights Reserved 58 From: R. Whiting,FDA-CFSAN

59. Copyrights © 2006 Silliker, Inc. All Rights Reserved 59 Summary Methods need to be chosen that deliver the most reliable, relevant and cost effective information to a process Rigorous but relatively simple,statistically valid protocols can be developed and run

60. Copyrights © 2006 Silliker, Inc. All Rights Reserved 60 Summary Economic factors must be evaluated along with technical merit Validations and verifications are a key part of the testing process Quality systems must be in place to monitor changes in the system There is no perfect method !