1. Verification of Performance Specifications
An Advanced View of Method Validation
Version 5.0, August 2012
Additional materials supplied for Advanced Assay Validation modules

2. Objectives Identify test classifications
Define what each validation experiment details for testing methods
Discuss what is recommended to perform each of the validation experiments for testing methods
Recognize how to evaluate data obtained from each of the validation experiments

3. Pre-Assessment Question #1
A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a:
High complexity, modified assay
Moderate complexity, unmodified assay
Food and Drug Administration (FDA)-approved, modified assay
Waived, FDA-approved, unmodified assay
Remember to pick the best answer.
The correct answer is: D. “Waived, FDA-approved, unmodified assay”

4. Pre-Assessment Question #2
The precision of a test method gives information related to the method’s:
Systematic error
Comparison of results to a reference method
Reproducibility
Likelihood of being affected by hemolysis, lipemia and icterus
Both A and B
Remember to pick the best answer.
The correct answer is: C. “Reproducibility”

5. Pre-Assessment Question #3
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? 20 18 16 15
Remember to pick the best answer.
The correct answer is: B. “18”

6. Pre-Assessment Question #4
Which linear regression equation component gives information regarding constant bias? y x
m (slope)
b (intercept)
Remember to pick the best answer.
The correct answer is: D. “b (intercept)”

7. Selecting a Method Evaluate diagnostic tests
Characteristics of testing methods
References: Technical literature and manufacturer’s information
Select method of analysis
Validate method performance
Implement method
Perform tests with appropriate Quality Control (QC) and External Quality Assurance (EQA)
Characteristics of testing methods
Applications: Types of specimens, sample volume recommended, time of performance, work, space recommended, etc.
Methodology: Sensitivity and specificity, reportable range, etc.
A number of factors are involved in the selection of methods and equipment required for testing (cost, space, access to local equipment support, etc.)
Important to review analyzers offered by different companies – do they meet the needs of your lab?
Need to validate the assay before you begin to use it to report patient test results.
Need to verify manufacturers claims; validate performance method

8. Method Validation What is method validation? Why must we validate?
When should we validate?
What is Method Validation?
Method validation and/or verification is the process by which a method is determined to be fit for purpose and intended use. Although method validation/verification are often used interchangeably, validation is usually performed on in-house and/or modified methods, while verification is taking a marketed/unmodified assay and verifying its performance.
Why is it important?
Different testing environment, need to demonstrate that the test method performs in your lab environment as the manufacturer states it should
Need to prove to yourself that the results reported by the manufacturer are reliable.
When?
Initially, before releasing patient results, and after any manufacturer changes/modifications and/or movement of the equipment.
What should we validate?

9. Method Validation (cont’d)
Why is validation important?
Division of Acquired Immunodeficiency Syndrome (DAIDS) requirement
How important is it that the results produced by the testing method are reliable?
Shouldn’t the laboratory know the level of performance of an adopted test method?
Validation is an FDA requirement under Investigational New Drug (IND).

10. Tests to Validate Waived Non-waived Unmodified FDA-approved
Waived tests are approved by the FDA for home use and by definition are simple to perform (i.e., pregnancy test) – do not require validation.
Non-waived tests included moderately and highly complex tests [DEFINE] – requires validation
Degree of validation/verification depends on status of the test method (FDA-approved/Non-FDA-approved/Modified/Unmodified)
Unmodified FDA-approved: using as intended by the manufacturer and licensed for use by the FDA
Modified or Non-FDA approved: using test kit for indications other than as intended by the manufacturer; not licensed for use by the FDA.
Unmodified FDA-approved
Modified and/or Non-FDA-approved

11. FDA Approval Resources
Vendor
Publications
Procedures/InVitroDiagnostics/LabTest/ucm htm
Sources for information regarding FDA approval status.

12. Skill Check
What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook?
Waived
Moderate
High
Remember to pick the best answer.
The correct answer is: B. “Moderate”
Moderate: No special pre-treatment steps recommended, performed on analyzer, very little or no interpretation required.

13. Skill Check
What would you consider to be the complexity of a rapid urine pregnancy assay?
Waived
Moderate
High
Remember to pick the best answer.
The correct answer is: A. “Waived”
Waived: Simple to perform, very little chance of error, can be performed outside of the lab by non-clinical personnel.

14. Skill Check
What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear?
Waived
Moderate
High
Remember to pick the best answer.
The correct answer is: C. “High”
High Complexity: Some degree of specimen preparation/pretreatment, interpretation/identification required.

15. Method Validation Before you begin:
Be sure you are familiar with the test method before starting
Know what to expect from the method (package insert, discussions with technical assistance, and field service representatives)
Do not include results outside of stated reportable ranges
Predict your findings; establish limits/evaluation criteria

16. Central Tendency Dispersion Terms for Discussion
Central Tendency – describes the way in which quantitative data tend to cluster around some value; If you run specimens again and again, results have a tendency to go to an average (mean)
Dispersion – spread of results

17. Terms for Discussion (cont’d)
Values
All results, even if they don’t look good.
Run 17

18. Error in Test Methods Some error is expected Examples
Error must be managed
Understanding
Defining specifications of allowable error
Measurement
Some error is expected; but need to manage error in order to report accurate results.
All methods have some level of systematic and random error.
The purpose of method validation is error assessment (i.e. random, systematic and total analytical errors)
During method validation a series of experiments are performed to estimate certain types of analytical errors:
Linearity experiments determine reportable range
Replication experiments estimate precision or random error
Comparison of methods experiments estimate accuracy or systematic error
Interference experiments estimate constant and proportional systematic errors ( or analytical specificity)
Detection limit experiments characterize analytical sensitivity.
We will review all of these concepts shortly.

19. Total Error of Testing System
Total Allowable Error
CLIA Guidelines per analyte
Other Guidelines
Systematic Error
Random Error
Total Error
Total Allowable Error: For a given test or assay, what we should expect to see when you combine systematic and random error.
CLIA guidelines have several indicators that can be used to help you determine your Total Allowable Error for a given test/assay.

20. Error Assessment Random Error (RE) Systematic Error (SE) Total Error
In one direction, cause results to be high or low
In either direction, unpredictable
Combined effect
Here we discuss the idea that Systematic + Random error = total error, and has the potential to push the result we get that distance from the actual true result. We can draw a picture to represent this.
Systematic Error: Tends towards or direction (either +/- from true value); size is stable and consistent; appears every time you perform the test.
Random Error: By nature is random; tends towards either direction (can add to or deduct from the true value); size is unpredictable. Average amount of random error occurs sporadically; each specimen going through the test will be affected to varying degrees.
CLIA guidelines available for estimation of Total Error; may not always be available for your particular assay.
David Rhodes – the most we should ever allow for Total Error is 30%.

21. Total Error Considerations
Low End Performance Standards
Recommendations derived from upper portion of reportable range are more difficult to achieve at lower concentrations
Maximum Total Error Allowed
Considered to be 30% by David Rhoads, except for amplification methods

22. Systematic and Random Errors
Systematic Error
Slope/Proportional error
Intercept/Constant error
Bias
Random Error
Mean
Standard deviation (SD)
Coefficient of variation (CV)
Systematic error is represented in terms of…
Random error is represented in terms of…

23. Tools for Use Data-Crunching Tools
Statistical calculators, graph paper
Spreadsheets with calculations
Here is where we describe and/or demonstrate tools that we have.
We can use the spreadsheets from the standards at this point…Or just free wheel it.
Need some data crunching tools at your disposal (i.e., Excel)
Validation software can be purchased (i.e., EP Evaluation, Analyze IT, etc.)
Westgard has free tools available for use online (
Validation Software (Westgard, Analyze-It, EP Evaluator)

24. How We Will Work Through This Module
One quantitative test taken through the validation process
One qualitative method taken through the validation process

25. Elements of Validation
Reportable Range
Precision
Accuracy
Elements of Validation
Reference Intervals
Sensitivity
The 6 Elements of Method Validation:
If FDA-approved/Unmodified – only Reportable Range, Precision, Accuracy and Reference Intervals need to be verified.
If Non-FDA approved/Modified – all 6 elements must be performed, including sensitivity and specificity.
Correction Factors:
Correction factors, if used, must be incorporated into the relevant test procedure and reflected in the appropriate Standard Operating Procedure (SOP).
Correction factors represent adjustments made to compensate for constant and proportional error (or bias).
Specificity

26. How we perform the testing
Precision
Definition: Reproducibility
Gives information related to random error
Introduction
20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)
Standard solutions
Control materials
Pools (short term only)
What is needed
Measure of the reproducibility of the assay.
Provides information related to random error.
How do you verify precision?
Must repeat testing on 20 samples over one day (short term) and over a period of 20 days (long term).
Within 20 days we hope to see good among of variation in terms of how the test performed and exposure to a variety of environmental conditions.
Repeat testing over short and long term (one day and 20 days, respectively)
How we perform the testing

27. Precision: How We Evaluate the Data
Calculate the following:
Mean
Standard deviation (SD)
Coefficient of Variation (CV)
What amount of random error is allowable, based on CLIA criteria?
Short term: 0.25 of allowable total error
Long term: 0.33 of allowable total error
First step, calculate mean, SD (standard deviation), and CV
CV = SD/Mean x 100% “the great leveler”
Compare information to manufacturers package insert, OR
Compare to CLIA recommendations for allowable random error:
CLIA criteria for allowable random error: no more than 25% for short term, or 33% for long term

28. Allowable Total Error Database
Link for:
Clinical Laboratory Improvement Amendments (CLIA)
College of American Pathologists (CAP)
Royal College of Pathologists of Australasia (RCPA)
Others

29. Precision: Levey-Jennings (LJ) Charts
Values
All results, even if they don’t look good.
Run 29

30. Precision: How We Evaluate the Data
How do we compare to manufacturer’s data?
Mean SD
CV: More commonly used, allows for easier comparison
Compare results to manufacturers data first, if comparable to manufacturers data you can indicate that the method is acceptable from a precision standard and move forward.
If not, then compare to CLIA.

31. Precision Example Mean of Level 1 Glucose 90 mg/dL
CLIA Total Allowable Error
6 mg/dL or ± 10%
Total Allowable Error Level 1 Glucose
0.1 x 90 = 9 mg/dL
Random error allowed:
0.25 x total allowable
0.33 x total allowable
As an example….
Calculated mean is 90 mg/mL
Per CLIA – 25% of mean is attributed to random error for Short Term (2.25 mg/dL); 33% for Long Term (2.97 mg/dL)
Compare CLIA calculations for short and long term? Do results make sense? Yes, would expect more random error over the long term.
Short-term precision
Long-term precision
0.25 x 9 mg/dL
0.33 x 9 mg/dL
2.25 mg/dL
2.97 mg/dL

32. Activity Work with Levey-Jennings graph and data
Work with mean and standard deviation to calculate a coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation
Determine if precision data is acceptable
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Precision Data Sets Worksheet, Graph paper

33. How we perform the testing
Accuracy
Definition: How close to the true value
Comparison of methods
Gives information related to systematic error
Potential conflicts on interpretation of results (reference values)
Introduction
40 different specimens
Cover reportable range of method
Quality versus quantity
What is needed
How does the new method compare to the reference method or standard?
Testing should be performed in duplicate over at least 5 days in order to account for analyzer variations over the specified time period.
Precision gives you information related to random error; Accuracy gives you information related to systematic error.
Duplicate measurements of each specimen on each method
Minimum of five days, prefer over 20 (since replicate testing is same)
How we perform the testing

34. Accuracy: How We Evaluate the Data
Graph the Data:
Difference plot
Real time
Comparison plot
Calculate
y = mx + b
Test method on Y-axis
b represents constant error
Reference (comparative) method on X-axis
m represents proportional error
Create graph electronically or by hand on an x/y axis.
Plot data, draw best fit line, calculate linear regression equation (y= mx +b).
b= y intercept, m= slope, x=comparative/reference method.
Best case scenario: Slope close to 1; Intercept close to 0 – would indicate that a result of 50 on current method is also 50 on comparative method.
Difference plot, if one-to-one relationship expected.
Shows analytical range of data, linearity of response over range and relationship between methods

35. Visual Inspection for Accuracy
(x1, y1)
Test Method
(x2, y2)
Slope = (y2- y1) / (x2- x1)
To evaluate accuracy, first run all of your samples on both the old instrument and the new instrument.
<<<click>>>
Next, plot the results you obtain. In this example the old instrument is along the X axis and the new instrument is along the Y axis.
Plot all of the results from your run.
Draw the “best fit” line.
Intercept
Reference Method 35

36. Accuracy: How We Evaluate the Data
Slope: Usually not significantly different from 1
Intercept: Not significantly different from 0
Significant difference with Medical Decision Points

37. Calculate Appropriate Statistics
Slope
Measure of proportional bias
m = (y1-y2)/(x1-x2) or “rise/run”
Slope greater than 1 means the Y (Test) values are generally higher than the X (Comparative) values
Slope of 1.11 means the Y (Test) values are on average 11% higher than the X (Comparative) values
Linear regression programs or calculator will perform this calculation for you.
Slope itself is a measure of proportional bias.
If m>1, the higher the value of x; the more variation on y.
If m>1, can assume y>x, i.e. test values for new method are generally higher than comparative method.
If m>1.11, test values are 11% greater than comparative method.
If m=0.95, test values are lower on average by 5%.

38. Calculate Appropriate Statistics (cont'd)
Intercept of the Line
Measure of constant bias between two methods
Y (Test) value at the point where the line crosses the Y axis
If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values
Assuming slope = 1, if y= 12, the test values are at least 12 units higher.
Assuming slope = 1, if y= -10, assume test values are 10 units lower than comparative method.

39. What type of bias do you see?
Accuracy
What type of bias do you see?
Answer on the next slide.

40. Accuracy (cont’d) Constant Bias Proportional Bias
Is this proportional or constant bias
Constant Bias: y intercept has changed
Proportional Bias: Slope has changed

41. Skill Check
Can a linear regression formula offer predictive value in relation to method comparisons?
Yes No
Remember to pick the best answer.
The correct answer is: A. “Yes”

42. Activity Create graph based on sample set
Determine slope from best-fit line
Determine Y-intercept from best-fit line
Explain the relationship between comparative and test results
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Comparison Data Set, Graph Paper

43. Reportable Range / Linearity
Definition: Lowest and highest test results that are reliable
Especially important with two point calibrations
Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR)
Introduction
Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)
Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens
At least four levels (five preferred)
What is needed
Linearity may be misnomer; hence the terminology of Analytical Measurement Range and derived Clinical Reportable Range.
How we perform the testing
CLSI recommends four measurements of each specimen; three are sufficient

44. Reportable Range: How We Evaluate the Data
Plot mean values of:
Measured values on Y-axis versus
Known or assigned values on X-axis
Visually inspect, draw best-fit line, estimate reportable range
Compare with expected values (typically provided by manufacturer)

45. Reportable Range Activity
Assigned
Value
Experimental Results
Average
Rep #1
Rep #2
Rep #3
Rep #4
10.0
____
11.0
100.0
99.0
103.0
101.0
300.0
303.0
305.0
304.0
306.0
500.0
505.0
506.0
800.0
740.0
741.0
744.0
742.0

46. Reportable Range Activity (cont'd)
Assigned
Value
Experimental Results
Average
Rep #1
Rep #2
Rep #3
Rep #4
10.0
10.5
11.0
100.0
101.5
99.0
103.0
101.0
300.0
304.5
303.0
305.0
304.0
306.0
500.0
505.5
505.0
506.0
800.0
741.8
740.0
741.0
744.0
742.0

47. Reportable Range Activity (cont'd)
Can everyone see the slight drop in linearity between 700 and 1,000?

48. AMR vs. CRR Analytical Measurement Range (AMR) Linearity
Clinically Reportable Range (CRR)
Discussion of definitions:
AMR: Straight linearity, minimum three (3) levels of known concentrations
CRR: Dilutions
NOTE: Validation materials must span the range of the Analytical Measurement Range (AMR); the matrix of the materials should not interfere with the method or bias the results.
Allows for dilution or other preparatory steps beyond routine

49. Skill Check
If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely?
AMR
CRR
Neither A or B
Both A and B
Remember to pick the best answer.
The correct answer is: A. “AMR”

50. Linearity Materials
Utilizing the marketing materials from the two chemistry linearity kits in your handouts:
Determine which kit would be more appropriate for use with the chemistry assay you chose earlier
Explain your reasoning
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): NOVA-ONE Chemistry Reference Kits, Fisher Healthcare (Microgenics CASCO DOCUMENT CALVER)
The first one is better…has an upper concentration of 750, more closely approximating the manufacturer’s linearity.

51. Graph Activity
Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide.
Review data; record any initial observations
Graph data on supplied graph paper
Determine your assay’s AMR
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Graph Paper

52. Linearity Experiment Results
Level
Rep 1
Rep 2
Rep 3
Rep 4 1 24 23 25 2
196
197
171
194 3
359
360
358
361 4
530
532
529
535 5
700
695
702
709
5 levels of known concentration linearity materials, run in duplicates of 4
What is the next step? Evaluate data – are there any outliers? Yes (171)
Statistically speaking you can only exclude one outlier
Exclude 171 and calculate the average

53. Activity
Using an Excel spreadsheet, create a graph and calculate linear regression statistics from the data provided
To what concentration have we proven linearity? (705 = AMR)

54. Lab's Average Known Conc 195.7 200 375 550 725
Rep 1
Rep 2
Rep 3
Rep 4
Lab's Average
Known
Conc 24 23 25
196
197
171
194
195.7
200
359
360
358
361
375
530
532
529
535
550
700
695
702
709
725
Exclude outlier (171) and calculate the average

56. Dilution Protocols
Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is 15 – 1400 mg/dL
Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Glucose Package Insert
If analyzer creates results greater than 1400, report as >1400
If less than 15, report as <15
Dilution Protocol: What will you use as a diluent? Consult the manufacturers package insert for guidance (i.e., saline)
Based on results, what is the maximum dilution we will ever perform?
We have proven AMR up to 705, and the highest reportable value is 1400, so at most dilute to 2X [2 x 700 = 1,400]

57. Reportable Results
Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results?
12 mg/dL
800 mg/dL
1600 mg/dL
Using the previously calculated values for AMR and CRR, and the dilution protocol, discuss how you would handle the following analyzer results

58. How we perform the testing
Reference Intervals
Definition: Normal range in healthy population
Used for diagnosis/clinical interpretation of results
Introduction
Pre-defined “normal” criteria for screening purposes
Transferring: 20 “normal” individuals’ specimens
Establishing: 120 “normal” individuals’ specimens
What is needed
For each reagent or kit, need to establish reference intervals to assist clinicians in interpretation of results.
Based on normal “healthy” population.
Test and compare to manufacturer: If all 20 specimens fall within the specified range, the reference ranges have been verified and you can adopt manufacturer suggestions and incorporate into your SOP. If not, need to establish reference ranges for your population (240 specimens: 120 each of male and female).
Another lab may have established reference ranges that can be used for verification purposes; must be documented.
How we perform the testing
Perform testing on all samples
Document results

59. Reference Intervals: How We Evaluate the Data
Transferring
Establishing
18 of 20 must fall within manufacturer’s ranges
Calculate mean and SD of data for each group
Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended)
If 3 of 20 fail…must look at another group of 20 for a total 40; now 36 must fall within the reference range.

60. Activity
Determine if assay is eligible for transference of reference intervals
Review a sample set of data to determine if transference may be performed; if not, determine next step(s)
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Normal Range Data, Glucose Package Insert
Adult Glucose Result:
Glucose manufacturer’s reference range (refer to package insert): mg/dL
Is the assay eligible for transference of reference intervals? NO (3 fall out of manufacturers reference range; 107, 106 and 110)
*Need to repeat with another 20, of which 36 of the 40 must fall within manufacturers expected range

61. How we perform the testing
Sensitivity
Definition: Lowest reliable value; lower limit of detection, especially of interest in drug testing and tumor markers
Different terminologies used by different manufacturers
Introduction
Blank solutions
Spiked samples
What is needed
Not required for FDA approved/ unmodified assay.
If FDA approved/Unmodified, information can be used directly and incorporated in the your SOP.
Need to adopt values in your SOP as evidence for sponsor/audit.
How we perform the testing
20 replicate measurements over short or long term, depending on focus

62. Sensitivity: How We Evaluate the Data
Three methods used:
Lower Limit of Detection (LLD):
Mean of the blank sample, plus two or three SD of blank sample
Biological Limit of Detection:
LLD plus two or three times SD of spiked sample with concentration of detection limit
Functional Sensitivity:
Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is reliable

63. Activity
Using the manufacturer’s package inserts, find the related information for sensitivity. How was it calculated?
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Glucose Package Insert
Referring to the package insert, determine what method was used to calculate sensitivity:
Limit of Detection (LOD) = 2.5mg/mL Limit of Quantitation (LOQ) = 5.0mg/mL
FDA approved/unmodified assay – can copy information directly into SOP and adopt values directly; do not need to validate

64. How we perform the testing
Specificity
Definition: Determination of how well a method measures the analyte of interest accompanied by potential interfering materials
Introduction
Standard solutions, participant specimens or pools
Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations
What is needed
Ability of your method to accurately measure an analyte within the presence of potential interfering substances.
Not required for FDA approved/ unmodified assay.
If FDA approved/Unmodified, adopt values in your SOP as evidence for sponsor/audit.
How we perform the testing
Duplicate measurements

65. Specificity: How We Evaluate the Data
Tabulate results for pairs of samples (dilution and interferent)
Calculate means for each (dilution and interferent)
Calculate the differences
Calculate the average interference of all specimens tested at a given concentration of interference
What else can cause interferences? Drugs/medication/hemolysis/lipemia.
Run assay with specimen diluted with blank (saline) and compare to same specimen diluted with potential interfering substance.

66. Qualitative Assays Compare diagnosis
Assume comparative (reference) method is accurate
Determine the following:
True Positives, True negatives
False Positives, False negatives
Calculate sensitivity and specificity and compare to manufacturer
Up until now, this module has spoken by and large to validation steps in relation to quantitative assays, or assays that generate a numeric result. What about qualitative assays, the tests that generate positive or negative, reactive or non-reactive type results? The steps recommended to validate these assays vary a bit in the respect that laboratories will compare “diagnoses” produced by the new testing method as compared to a current or reference method.
The rule of thumb here is to assume the reference method (current method) is accurate, and if there is a discrepancy in the results; the new method’s result is the one in question. Given this convention, the laboratory can tally discrepant results into classifications of false positives or false negatives, as compared with the reference method. The results that compare across methods would be tallied as true positives or negatives. Given this data, the laboratory can calculate a comparative sensitivity and specificity, that allows evaluation of the new method based on published information from the new method’s manufacturer. 66

67. Qualitative Assays: Control of Validation
Negative and Positive Quality Controls
Use QC materials recommended by manufacturer for verification purposes
Determine validity of other results, e.g., method comparisons
Evaluate failed runs if they occur during verification process

68. Qualitative Methods: Precision
How is it performed?
Runs of specimens with analyte concentrations near the cutoff point
Three specimens, one at cutoff, one just below cutoff, and one just above cutoff (± 20% recommended)
Replicate measurements of each of three specimens (20 each, minimum)
How is it evaluated?
Determine percentage of positives and negatives for each specimen
Evaluate cutoff, as well as other two specimens

69. Accuracy/Method Comparisons
How is it performed?
Specimens typical of population (to be tested in future use of method)
50 positive specimens and 50 negative specimens recommended; minimum 20 each
Performed over 10 to 20 days
How is it evaluated?
Discrepant results near cutoff?
Most often sensitivity and specificity used to describe performance

70. Comparative or Reference Method Result Positive Predictive Value
Qualitative Methods
Comparative or Reference Method Result
Positive
Negative
Test Method Result
True Positive
False Positive
Positive Predictive Value
False Negative
True Negative
Negative Predictive
Value
Sensitivity
Specificity
True vs. False
False Positive Rate - False Positives divided by total number of Negatives
False Negative Rate - False Negatives divided by total number of Positives

71. Qualitative Methods (cont'd)
Comparative or Reference Method Result
Positive
Negative
Test Method Result
True Positive
False Positive
Positive Predictive Value
False Negative
True Negative
Negative Predictive
Value
Sensitivity
Specificity
Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)
Specificity = 100 x True Negatives divided by (True Negatives + False Positives)

72. Qualitative Methods (cont'd)
Comparative or Reference Method Result
Positive
Negative
Test Method Result
True Positive
False Positive
Positive Predictive Value
False Negative
True Negative
Negative Predictive
Value
Sensitivity
Specificity
Predictive Values - Operation of a test on a mixed population of Positive and Negatives
A property of the test and the population; and affected by prevalence of Positives
Positive Predictive Value = True Positives divided by (True Positives + False Positives)
Negative Predictive Value = True Negatives divided by (True Negatives + False Negatives)

73. Evaluation Criteria High Diagnostic Value 100% Sensitivity
100% Specificity
What happens if True Positive rate is equal to the False Positive rate?
Sensitivity = TP/(TP+FN) = must have zero false negatives (FN) to achieve 100%
Specificity = TN/(TN+FP) = must have zero false positives (FP) to achieve 100%
What happens if we have 50% sensitivity and 50% specificity? Would have equal values for TN and FP; and equal values for TP and FN – does that assay have real diagnostic value? NO

74. Activity
Estimate sensitivity and specificity of a qualitative method given a data set.
<Refer to supplemental materials provided for this section>
Supplemental Materials Needed (Advanced Method Validation): Qualitative Method Comparison Data
Use the data to tally the TP, FP, FN, TN; then calculate the sensitivity and specificity:
[TP=8, FP=2, FN=1, TN=9]
Sensitivity = TP/(TP+FN) = 89%
Specificity = TN/(TN+FP) = 82%
Is this acceptable performance for your lab? Methods vary in terms of specificity and sensitivity.

75. Activity (cont’d)
Create a validation plan for a quantitative assay to be performed in your laboratory.

76. In Closing
Now that you have completed this module, you should be able to:
Identify test classifications
Define what each validation experiment details for testing methods
Discuss what is recommended to perform each of the validation experiments for testing methods
Recognize how to evaluate data obtained from each of the validation experiments

77. Post-Assessment Question #1
A rapid HIV test would likely be classified as a:
High complexity, modified assay
Moderate complexity, unmodified assay
FDA-approved, modified assay
Waived, FDA-approved, unmodified assay
Remember to pick the best answer.
The correct answer is: D. “Waived, FDA-approved, unmodified assay”

78. Post-Assessment Question #2
The precision of a test method gives information related to the method’s:
Systematic error
Comparison of results to a reference method
Reproducibility
Likelihood of being affected by hemolysis, lipemia and icterus
Both A and B
Remember to pick the best answer.
The correct answer is: C. “Reproducibility”

79. Post-Assessment Question #3
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? 20 18 16 15
Remember to pick the best answer.
The correct answer is: B. “18”

80. Post-Assessment Question #4
Which linear regression equation component gives information regarding constant bias? y x
m (slope)
b (intercept)
Remember to pick the best answer.
The correct answer is: D. “b (intercept)”

81. References DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines.
Validation of Qualitative Methods.
42 CFR §
College of American Pathologists Commission on Laboratory Accreditation, Accreditation Checklists, April 2006.
Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard QC, Inc., 2003.
Clinical and Laboratory Standards Institute. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002.
Clinical and Laboratory Standards Institute. Evaluation of Precision.
Performance of Quantitative Measurement Methods. NCCLS document EP5-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004.
Clinical and Laboratory Standards Institute. User verification of Performance for Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005.

82. Wrap Up