1. Validation: concept, & considerations
Beni Kaufman

2. Will be presenting: Review Process vs. Modular validation The concept
Validation Components and their measurement
experimental design of PCR validation
Process vs. Modular validation

3. References: Guidance for Industry: Bioanalytical Method Validation.
U.S. Department of Health and Human Services, Food and Drug Administration (FDA), Center for Drug Evaluation and Research (CDER), Center for Veterinary Medicine (CVM) May 2001
PCR Validation & Performance Characteristics
Analytical Environmental Immunochemical Consortium (AEIC) Biotech Consensus Paper; S. Charlton, R. Giroux, D. Hondred, C. Lipton, K. Worden
Validation of Analytical procedures: Methodology, International Conference on Harmonization of Technical Requirements for the Registration of Pharmaceuticals for Human Use, 1996

4. Politically!!! sensitive material!! Warning
Discuses components… avoid criteria!
Discuses components… avoid criteria!

5. Validation Components

6. Selectivity (Specificity)
The ability of the analytical method to differentiate (and quantify) the analyte in the presence of other components in the sample (to amplify only the Sequence of interest.)
Selectivity may be affected by:
Interference:
Cross amplification of non target sequences (function of, Primer design)
Matrix effects:
Background signal (Sybr green)
Quality & quantity of DNA
Reaction conditions (master-mix, thermocycling profile)
For selectivity, analyses of blank samples of the appropriate biological matrix (plasma, urine, or other matrix) should be obtained from at least six sources. Each blank sample should be tested for interference, and selectivity should be ensured at the lower limit of quantification (LLOQ). If the method is intended to quantify more than one analyte, each analyte should be tested to ensure that there is no interference.

7. Selectivity (Cont.) Assessed by:
Fragment length analysis (right size amplicon)
Electrophoresis gel analysis CE
Show gel, and melting curve

8. Selectivity (Cont.) Dissociation Curve Assessed by:
do not use r2774, , 15Hr 58Min.mxp
Assessed by:
Melting curve analysis

9. Precision Variation among rep.s within an assay Same as Repeatability
The closeness of agreement (degree of scatter) between a series of measurements obtained from multiple sampling of the same homogeneous sample under the prescribed conditions.
Variation among rep.s within an assay
Same as Repeatability
Measured by parameters of variation, mostly %CV
Precision should be measured using a minimum of five determinations per concentration. A minimum of three concentrations in the range of expected concentrations is recommended. The precision determined at each concentration level should not exceed 15% of the coefficient of variation (CV) except for the LLOQ, where it should not exceed 20% of the CV. Precision is further subdivided into within-run, intra-batch precision or repeatability, which assesses precision during a single analytical run, and between-run, interbatch precision or repeatability, which measures precision with time, and may involve different analysts, equipment, reagents, and laboratories.

10. Precision parameters:
Sample
Result
Average
Variance
Standard Deviation
%CV
=AVERAGE(D5:D8)
=VAR(D5:D8)
=STDEV(D5:D8)
=100*(StDev/Average)
1.1
0.12
0.1475
1.2
0.24
1.3
0.11
1.4
2.1
0.3
0.31
2.2
0.34
2.3
0.29
2.4
3.1
0.52
0.48
0.003
3.2
0.51
3.3
0.49
3.4
0.4

11. Accuracy /Trueness
The closeness of mean test results to the true value of the analyte.
Qualitative assay:
Measured by error rate:
% false positive = False positives/ # of negatives
% false negative = False negatives/# of positives
Accuracy is determined by replicate analysis of samples containing known amounts of the analyte. Accuracy should be measured using a minimum of five determinations per concentration. A minimum of three concentrations in the range of expected concentrations is recommended. The mean value should be within 15% of the actual value except at LLOQ, where it should not deviate by more than 20%. The deviation of the mean from the true value serves as the measure of accuracy.

12. Accuracy/Trueness (cont.)
Quantitative assay:
The mean recovery at several points across the quantitative range
% Recovery =100 (observed/actual)
(also,
the deviation of the mean from the true value)

13. Accuracy/Trueness measured
level (%)
Sample
Result
%Recovery
Average Recovery
0.1
sample 1
0.12
120
147.5
sample 2
0.24
240
sample 3
0.11
110
sample 4
0.3
100
0.34
0.29
0.31
0.5
0.52
104 96
0.51
102
0.49 98
0.4 80

14. Linearity & Range
Linearity: The ability of the assay (within a given range) to obtain test results which are directly proportional to the concentration/amount of the analyte
Range: The interval between the upper & lower concentrations of an analyte for which the assay has suitable levels of precision, accuracy & linearity.

15. Linearity & Range (cont.)
Linearity and Range can be evaluated simultaneously
Demonstrated on a dilution series (transgene genomic DNA/null genomic DNA) across a relevant range of concentrations
The Range is established by confirming acceptable degrees of linearity, accuracy, & precision, within or at the extremes of a specified range.

16. Linearity evaluated
Linearity is evaluated by a plot of signals as a function of analyte concentration & linear regression analysis.

18. Sensitivity Two concepts of sensitivity:
Change in response per amount of reactant -> dose-response curve
In PCR the dose response is derived from the amplification efficiency - We optimize the assay for a maximal dose response (~100% amp. Efficiency)
Therefore,
dose-response is reflected in the standard curve. It’s captured in the Linearity component &
is the basis for quantification
The source of our resolution power

19. Sensitivity (cont.)
Limit of detection (LOD), The minimum amount of target analyte that can be detected with a given level of confidence
Applies to QL & QT PCR
Limit of quantification (LOQ), The lowest amount of target analyte that can be quantified with acceptable levels of precision and accuracy.
Applies only to QT PCR

20. Determining LOD & LOQ: “Spiking” series:
Decreasing amounts of transgenic seed are mixed in with conventional seed to create a series of seed pools with varying proportion of transgenes.
Seed pools are ground to flour
DNA isolated from flour and used for PCR; targeting the corresponding target sequence.

21. Sensitivity (cont.)
The LOD will be lowest spike detected with an acceptable confidence level.
The LOQ will be the lowest spike that can be differentiated from zero with an acceptable confidence level

22. Ruggedness
The effectiveness of an analytical process in face of small environmental/operating conditions, such as:
Different analysts
Different equipment
Different labs
Effectiveness is measured as changes in the precision or accuracy.

23. Ruggedness (cont.)
Effectiveness is measured as changes in the precision or accuracy:
For qualitative PCR evaluated by the changes in error rate and LOD
For quantitative PCR evaluated by HORRAT
Where the Relative Standard Deviation of Reproducibility (RSDr) is given as:
RSDr = 2(1-0.5lnC) ~ 2C
(C= concentration or quantity) And
HORRAT = RSDr(observed)/RSDr(expected)
HORRAT is expected to be close to 1
Horwitz, W. (1995) Protocol for the design, conduct and interpretation of method performance studies, Pure and Appl. Chem, 67:

24. Ruggedness measured 1.045797786 Result (%) RSDr Obs RSDr Exp Spike (%)
HORRAT
=2(1-0.5lnResult)
=2(1-0.5lnTrue)
=RSDr obs/RSDr exp
0.1
0.1475
0.086
0.303
0.3
0.31
0.829
0.796
0.5
0.48
1.266
1.307
0.6
0.6375
2.775
1.489 1
1.15
2.14 2
1.07
1.5
1.65
2.501
2.405
2.2
2.55
2.936
2.788

25. Robustness
Describes the reliability of an analysis with respect to variations in method parameters.
Measured by experimentally defining the critical range of:
Template concentration
Primer concentration
Mg2 Concentration
Thermocycling temperature range
Usually part of the assay optimization, prior to the validation process.

26. Cartoon Break

27. Seems to be a tedious process!It
Is !!!

28. Can take away some of the edge…
But,
the right
experimental design
Can take away some of the edge…
For example:

29. QT PCR Validation design:
Experiment:
Series of conventional seed pools fortified with transgenic seed at a decreasing ratio.
(For example: from 2% to 0.01% at -0.5X increments).
Highest level serves as positive control
Negative control
Five reps per level
Isolate, quantify, normalize, PCR (IQNP)
All in all: 8 spike levels x 5 replicates = 40 amplifications
Repeat 3 times, 3 different instruments, different analysts,
(3 different dates (?) Astrological effect)

30. QT PCR Validation design:
Analyze
Selectivity: all amplifications yielded the right size amplicon (on gel, or by Tm)
Precision: Calculate %CV among reps within plates
Accuracy: Calculate mean % recovery within plates
Linearity: use samples as standards – create standard curve- test linearity
Range: based on results of Precision, Accuracy, & Linearity; define range.
LOD: Identify the lowest detected spike with an accepted confidence limit
LOQ: Identify lowest spike that its confidence interval does not overlap zero.
Ruggedness: HORRAT, or alternatively, ANOVA between plates, runs, annalists.

31. QL PCR Validation design
Experiment
Series of conventional seed pools fortified with transgenic seed at a decreasing ratio.
(… from 2% to 0.01% at -0.5X increments).
Highest level serves as positive control
Negative control
Five reps per level
IQNP
Analyze:
Selectivity: all amplifications yielded the right size amplicon (On gel or by Tm)
LOD: The lowest spike level to yield amplification = tentative LOD

32. QL PCR Validation design
Experiment:
Two plates, each plate, half null, and half spiked at tentative LOD.
Isolate, quantify, normalize,
PCR the two plates on different instruments, different analysts, etc
Analyze:
Accuracy: Calculate positive and negative error rate.
Confirm LOD: if %false negative < defined criteria (5?)
Ruggedness: compare error rates between plates/instruments/analysts

33. That wasn’t that bad wasn’t it?

34. Not only PCR!
The testing process is made of a number of consecutive steps, all can be validated, some have to be validated
Sampling
Sub-sampling
DNA Isolation
DNA Quantification
DNA Normalization
PCR
Post-PCR
Data Analysis

35. Brought about the idea of Modular Validation
The recognition that many of the applications – steps, in the testing process require independent validation of their function
&
For better efficiency
Brought about the idea of Modular Validation
A. Holst-Jensen, J-AOAC, 1995

36. Validate each step (module).
Modular Validation
Validate each step (module).
Once, validated, different modules can be combined in to a process that no longer require validation
DNA is DNA!?
IT IS NOT.
For example: PCR reaction is validated using plasmids as standards
DNA isolation protocol from flour is validated
The two combind.

37. Whole Process Validation
Particle size
DNA isolation efficiency
Instrument error
Matrix effect
Standards
All affect the out come of the testing process, therefore, the validation is of the whole process and only in the context of the given matrix, instrumentation, & standards…

38. Any deviation…will require
You can’t “mix & match”
Any deviation…will require
VALIDATION.