Download presentation

Presentation is loading. Please wait.

Published byMarcel Isaac Modified about 1 year ago

1
This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS). Verification of Performance Specifications An Advanced View of Method Validation Version 5.0, August 2012

2
Identify test classifications Define what each validation experiment details for testing methods Discuss what is recommended to perform each of the validation experiments for testing methods Recognize how to evaluate data obtained from each of the validation experiments Objectives 2

3
A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a: A.High complexity, modified assay B.Moderate complexity, unmodified assay C.Food and Drug Administration (FDA)-approved, modified assay D.Waived, FDA-approved, unmodified assay Pre-Assessment Question #1 3

4
The precision of a test method gives information related to the method’s: A.Systematic error B.Comparison of results to a reference method C.Reproducibility D.Likelihood of being affected by hemolysis, lipemia and icterus E.Both A and B Pre-Assessment Question #2 4

5
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? A.20 B.18 C.16 D.15 Pre-Assessment Question #3 5

6
Which linear regression equation component gives information regarding constant bias? A.y B.x C.m (slope) D.b (intercept) Pre-Assessment Question #4 6

7
Selecting a Method 7 Evaluate diagnostic tests Characteristics of testing methods References: Technical literature and manufacturer’s information Select method of analysis Validate method performance Implement method Perform tests with appropriate Quality Control (QC) and External Quality Assurance (EQA)

8
Method Validation Why must we validate? When should we validate? What should we validate? 8 What is method validation?

9
Why is validation important? Division of Acquired Immunodeficiency Syndrome (DAIDS) requirement How important is it that the results produced by the testing method are reliable? Shouldn’t the laboratory know the level of performance of an adopted test method? Method Validation (cont’d) 9

10
Tests to Validate Waived Non-waived Unmodified FDA-approved Modified and/or Non-FDA-approved 10

11
Vendor Publications http://www.fda.gov/MedicalDevices/ProductsandMedical Procedures/InVitroDiagnostics/LabTest/ucm126079.htm FDA Approval Resources 11

12
What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook? A.Waived B.Moderate C.High Skill Check 12

13
What would you consider to be the complexity of a rapid urine pregnancy assay? A.Waived B.Moderate C.High Skill Check 13

14
What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear? A.Waived B.Moderate C.High Skill Check 14

15
Before you begin: Be sure you are familiar with the test method before starting Know what to expect from the method (package insert, discussions with technical assistance, and field service representatives) Do not include results outside of stated reportable ranges Predict your findings; establish limits/evaluation criteria Method Validation 15

16
Terms for Discussion 16 Central Tendency Dispersion

17
Terms for Discussion (cont’d) 17 Values Run

18
Some error is expected Examples Error must be managed Understanding Defining specifications of allowable error Measurement Error in Test Methods 18

19
Total Error of Testing System 19 CLIA Guidelines per analyte Other Guidelines Systematic Error Random Error Total Error

20
Error Assessment 20 In either direction, unpredictable In one direction, cause results to be high or low Combined effect Systematic Error (SE) Random Error (RE) Total Error (TE)

21
Low End Performance Standards Recommendations derived from upper portion of reportable range are more difficult to achieve at lower concentrations Maximum Total Error Allowed Considered to be 30% by David Rhoads, except for amplification methods Total Error Considerations 21

22
Systematic Error Slope/Proportional error Intercept/Constant error Bias Random Error Mean Standard deviation (SD) Coefficient of variation (CV) Systematic and Random Errors 22

23
Tools for Use 23 Spreadsheets with calculations Validation Software (Westgard, Analyze- It, EP Evaluator) Statistical calculators, graph paper Data- Crunching Tools

24
One quantitative test taken through the validation process One qualitative method taken through the validation process How We Will Work Through This Module 24

25
Reportable Range Precision Accuracy Reference Intervals Sensitivity Specificity Elements of Validation 25

26
Repeat testing over short and long term (one day and 20 days, respectively) 20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL) Standard solutions Control materials Pools (short term only) Precision Definition: Reproducibility Gives information related to random error Introduction What is needed How we perform the testing 26

27
Precision: How We Evaluate the Data 27 Mean Standard deviation (SD) Coefficient of Variation (CV) Short term: 0.25 of allowable total error Long term: 0.33 of allowable total error Calculate the following: What amount of random error is allowable, based on CLIA criteria?

28
Link for: Clinical Laboratory Improvement Amendments (CLIA) College of American Pathologists (CAP) Royal College of Pathologists of Australasia (RCPA) Others http://www.dgrhoads.com/db2004/ae2004.php Allowable Total Error Database 28

29
Precision: Levey-Jennings (LJ) Charts 29 Values Run

30
Precision: How We Evaluate the Data 30 Mean SD CV: More commonly used, allows for easier comparison How do we compare to manufacturer’s data?

31
Precision Example 31 Mean of Level 1 Glucose CLIA Total Allowable Error Total Allowable Error Level 1 Glucose Random error allowed: 90 mg/dL 6 mg/dL or ± 10% 0.1 x 90 = 9 mg/dL 0.25 x total allowable 0.25 x 9 mg/dL 2.25 mg/dL 0.33 x total allowable 0.33 x 9 mg/dL 2.97 mg/dL Long-term precision Short-term precision

32
Work with Levey-Jennings graph and data Work with mean and standard deviation to calculate a coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation Determine if precision data is acceptable Activity 32

33
Accuracy Definition: How close to the true value Comparison of methods Gives information related to systematic error Potential conflicts on interpretation of results (reference values) Introduction 40 different specimens Cover reportable range of method Quality versus quantity What is needed Duplicate measurements of each specimen on each method Minimum of five days, prefer over 20 (since replicate testing is same) How we perform the testing 33

34
Accuracy: How We Evaluate the Data Graph the Data: Test method on Y- axis Reference (comparative) method on X-axis Shows analytical range of data, linearity of response over range and relationship between methods Real time Difference plot Comparison plot Calculate y = mx + b b represents constant error m represents proportional error 34

35
Visual Inspection for Accuracy 35 Test Method Reference Method Intercept (x 1, y 1 ) (x 2, y 2 ) Slope = (y 2 - y 1 ) / (x 2 - x 1 )

36
Slope: Usually not significantly different from 1 Intercept: Not significantly different from 0 Significant difference with Medical Decision Points Accuracy: How We Evaluate the Data 36

37
Slope Measure of proportional bias m = (y1-y2)/(x1-x2) or “rise/run” Slope greater than 1 means the Y (Test) values are generally higher than the X (Comparative) values Slope of 1.11 means the Y (Test) values are on average 11% higher than the X (Comparative) values Calculate Appropriate Statistics 37

38
Intercept of the Line Measure of constant bias between two methods Y (Test) value at the point where the line crosses the Y axis If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values Calculate Appropriate Statistics (cont'd) 38

39
Accuracy 39 What type of bias do you see?

40
Accuracy (cont’d) Constant BiasProportional Bias 40

41
Can a linear regression formula offer predictive value in relation to method comparisons? A.Yes B.No Skill Check 41

42
Create graph based on sample set Determine slope from best-fit line Determine Y-intercept from best-fit line Explain the relationship between comparative and test results Activity 42

43
CLSI recommends four measurements of each specimen; three are sufficient Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets) Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens At least four levels (five preferred) Reportable Range / Linearity Definition: Lowest and highest test results that are reliable Especially important with two point calibrations Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR) Introduction What is needed How we perform the testing 43

44
Reportable Range: How We Evaluate the Data 44 Measured values on Y-axis versus Known or assigned values on X-axis Plot mean values of: Compare with expected values (typically provided by manufacturer) Visually inspect, draw best-fit line, estimate reportable range

45
Reportable Range Activity Assigned Value Experimental Results AverageRep #1Rep #2Rep #3Rep #4 10.0____11.010.011.010.0 100.0____99.0103.0 101.0 300.0____303.0305.0304.0306.0 500.0____505.0506.0505.0506.0 800.0____740.0741.0744.0742.0 45

46
Reportable Range Activity (cont'd) Assigned Value Experimental Results AverageRep #1Rep #2Rep #3Rep #4 10.010.511.010.011.010.0 100.0101.599.0103.0 101.0 300.0304.5303.0305.0304.0306.0 500.0505.5505.0506.0505.0506.0 800.0741.8740.0741.0744.0742.0 46

47
Reportable Range Activity (cont'd) 47

48
AMR vs. CRR Analytical Measurement Range (AMR) Linearity Clinically Reportable Range (CRR) Allows for dilution or other preparatory steps beyond routine 48

49
If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely? A.AMR B.CRR C.Neither A or B D.Both A and B Skill Check 49

50
Utilizing the marketing materials from the two chemistry linearity kits in your handouts: 1.Determine which kit would be more appropriate for use with the chemistry assay you chose earlier 2.Explain your reasoning Linearity Materials 50

51
Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide. 1.Review data; record any initial observations 2.Graph data on supplied graph paper 3.Determine your assay’s AMR Graph Activity 51

52
LevelRep 1Rep 2Rep 3Rep 4 124232524 2196197171194 3359360358361 4530532529535 5700695702709 Linearity Experiment Results 52

53
Activity Using an Excel spreadsheet, create a graph and calculate linear regression statistics from the data provided 53

54
Rep 1Rep 2Rep 3Rep 4 Lab's Average Known Conc 24232524 25 196197171194195.7200 359360358361360375 530532529535532550 700695702709702725 54

55
55

56
Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is 15 – 1400 mg/dL Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP Dilution Protocols 56

57
Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results? 1.12 mg/dL 2.800 mg/dL 3.1600 mg/dL Reportable Results 57

58
Reference Intervals 58 Definition: Normal range in healthy population Used for diagnosis/clinical interpretation of results Introduction Pre-defined “normal” criteria for screening purposes Transferring: 20 “normal” individuals’ specimens Establishing: 120 “normal” individuals’ specimens What is needed Perform testing on all samples Document results How we perform the testing

59
TransferringEstablishing 18 of 20 must fall within manufacturer’s ranges Calculate mean and SD of data for each group Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended) 59 Reference Intervals: How We Evaluate the Data

60
Activity 60 Determine if assay is eligible for transference of reference intervals Review a sample set of data to determine if transference may be performed; if not, determine next step(s)

61
Sensitivity Definition: Lowest reliable value; lower limit of detection, especially of interest in drug testing and tumor markers Different terminologies used by different manufacturers Introduction Blank solutions Spiked samples What is needed 20 replicate measurements over short or long term, depending on focus How we perform the testing 61

62
Sensitivity: How We Evaluate the Data Lower Limit of Detection (LLD): Mean of the blank sample, plus two or three SD of blank sample Biological Limit of Detection: LLD plus two or three times SD of spiked sample with concentration of detection limit Biological Limit of Detection: LLD plus two or three times SD of spiked sample with concentration of detection limit Functional Sensitivity: Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is reliable Functional Sensitivity: Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is reliable Three methods used: 62

63
Activity 63 Using the manufacturer’s package inserts, find the related information for sensitivity. How was it calculated?

64
Specificity Definition: Determination of how well a method measures the analyte of interest accompanied by potential interfering materials Introduction Standard solutions, participant specimens or pools Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations What is needed Duplicate measurements How we perform the testing 64

65
Specificity: How We Evaluate the Data Tabulate results for pairs of samples (dilution and interferent) Calculate means for each (dilution and interferent) Calculate the differences Calculate the average interference of all specimens tested at a given concentration of interference 65

66
Compare diagnosis Assume comparative (reference) method is accurate Determine the following: True Positives, True negatives False Positives, False negatives Calculate sensitivity and specificity and compare to manufacturer Qualitative Assays 66

67
Negative and Positive Quality Controls Use QC materials recommended by manufacturer for verification purposes Determine validity of other results, e.g., method comparisons Evaluate failed runs if they occur during verification process Qualitative Assays: Control of Validation 67

68
How is it performed? Runs of specimens with analyte concentrations near the cutoff point Three specimens, one at cutoff, one just below cutoff, and one just above cutoff (± 20% recommended) Replicate measurements of each of three specimens (20 each, minimum) How is it evaluated? Determine percentage of positives and negatives for each specimen Evaluate cutoff, as well as other two specimens Qualitative Methods: Precision 68

69
How is it performed? Specimens typical of population (to be tested in future use of method) 50 positive specimens and 50 negative specimens recommended; minimum 20 each Performed over 10 to 20 days How is it evaluated? Discrepant results near cutoff? Most often sensitivity and specificity used to describe performance Accuracy/Method Comparisons 69

70
Qualitative Methods Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 70 False Positive Rate - False Positives divided by total number of Negatives False Negative Rate - False Negatives divided by total number of Positives True vs. False

71
Qualitative Methods (cont'd) Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 71 Sensitivity = 100 x True Positives divided by (True Positives + False Negatives) Specificity = 100 x True Negatives divided by (True Negatives + False Positives)

72
Qualitative Methods (cont'd) Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 72 Predictive Values - Operation of a test on a mixed population of Positive and Negatives A property of the test and the population; and affected by prevalence of Positives Positive Predictive Value = True Positives divided by (True Positives + False Positives) Negative Predictive Value = True Negatives divided by (True Negatives + False Negatives)

73
High Diagnostic Value 100% Sensitivity 100% Specificity What happens if True Positive rate is equal to the False Positive rate? Evaluation Criteria 73

74
Activity 74 Estimate sensitivity and specificity of a qualitative method given a data set.

75
Activity (cont’d) 75 Create a validation plan for a quantitative assay to be performed in your laboratory.

76
Now that you have completed this module, you should be able to: Identify test classifications Define what each validation experiment details for testing methods Discuss what is recommended to perform each of the validation experiments for testing methods Recognize how to evaluate data obtained from each of the validation experiments In Closing 76

77
A rapid HIV test would likely be classified as a: A.High complexity, modified assay B.Moderate complexity, unmodified assay C.FDA-approved, modified assay D.Waived, FDA-approved, unmodified assay Post-Assessment Question #1 77

78
The precision of a test method gives information related to the method’s: A.Systematic error B.Comparison of results to a reference method C.Reproducibility D.Likelihood of being affected by hemolysis, lipemia and icterus E.Both A and B Post-Assessment Question #2 78

79
When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? A.20 B.18 C.16 D.15 Post-Assessment Question #3 79

80
Which linear regression equation component gives information regarding constant bias? A.y B.x C.m (slope) D.b (intercept) Post-Assessment Question #4 80

81
DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines. www.westgard.com. www.westgard.com Validation of Qualitative Methods. 42 CFR § 493.1253. College of American Pathologists Commission on Laboratory Accreditation, Accreditation Checklists, April 2006. Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard QC, Inc., 2003. Clinical and Laboratory Standards Institute. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002. Clinical and Laboratory Standards Institute. Evaluation of Precision. Performance of Quantitative Measurement Methods. NCCLS document EP5-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004. Clinical and Laboratory Standards Institute. User verification of Performance for Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005. References 81

82
Wrap Up 82

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google