Presentation is loading. Please wait.

Presentation is loading. Please wait.

This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,

Similar presentations


Presentation on theme: "This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,"— Presentation transcript:

1 This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS). Verification of Performance Specifications An Advanced View of Method Validation Version 5.0, August 2012

2  Identify test classifications  Define what each validation experiment details for testing methods  Discuss what is recommended to perform each of the validation experiments for testing methods  Recognize how to evaluate data obtained from each of the validation experiments Objectives 2

3 A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a: A.High complexity, modified assay B.Moderate complexity, unmodified assay C.Food and Drug Administration (FDA)-approved, modified assay D.Waived, FDA-approved, unmodified assay Pre-Assessment Question #1 3

4 The precision of a test method gives information related to the method’s: A.Systematic error B.Comparison of results to a reference method C.Reproducibility D.Likelihood of being affected by hemolysis, lipemia and icterus E.Both A and B Pre-Assessment Question #2 4

5 When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? A.20 B.18 C.16 D.15 Pre-Assessment Question #3 5

6 Which linear regression equation component gives information regarding constant bias? A.y B.x C.m (slope) D.b (intercept) Pre-Assessment Question #4 6

7 Selecting a Method 7  Evaluate diagnostic tests  Characteristics of testing methods  References: Technical literature and manufacturer’s information  Select method of analysis  Validate method performance  Implement method  Perform tests with appropriate Quality Control (QC) and External Quality Assurance (EQA)

8 Method Validation Why must we validate? When should we validate? What should we validate? 8 What is method validation?

9  Why is validation important?  Division of Acquired Immunodeficiency Syndrome (DAIDS) requirement  How important is it that the results produced by the testing method are reliable?  Shouldn’t the laboratory know the level of performance of an adopted test method? Method Validation (cont’d) 9

10 Tests to Validate Waived Non-waived Unmodified FDA-approved Modified and/or Non-FDA-approved 10

11  Vendor  Publications  Procedures/InVitroDiagnostics/LabTest/ucm htm FDA Approval Resources 11

12 What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook? A.Waived B.Moderate C.High Skill Check 12

13 What would you consider to be the complexity of a rapid urine pregnancy assay? A.Waived B.Moderate C.High Skill Check 13

14 What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear? A.Waived B.Moderate C.High Skill Check 14

15  Before you begin:  Be sure you are familiar with the test method before starting  Know what to expect from the method (package insert, discussions with technical assistance, and field service representatives)  Do not include results outside of stated reportable ranges  Predict your findings; establish limits/evaluation criteria Method Validation 15

16 Terms for Discussion 16 Central Tendency Dispersion

17 Terms for Discussion (cont’d) 17 Values Run

18  Some error is expected  Examples  Error must be managed  Understanding  Defining specifications of allowable error  Measurement Error in Test Methods 18

19 Total Error of Testing System 19 CLIA Guidelines per analyte Other Guidelines Systematic Error Random Error Total Error

20 Error Assessment 20 In either direction, unpredictable In one direction, cause results to be high or low Combined effect Systematic Error (SE) Random Error (RE) Total Error (TE)

21  Low End Performance Standards  Recommendations derived from upper portion of reportable range are more difficult to achieve at lower concentrations  Maximum Total Error Allowed  Considered to be 30% by David Rhoads, except for amplification methods Total Error Considerations 21

22  Systematic Error  Slope/Proportional error  Intercept/Constant error  Bias  Random Error  Mean  Standard deviation (SD)  Coefficient of variation (CV) Systematic and Random Errors 22

23 Tools for Use 23 Spreadsheets with calculations Validation Software (Westgard, Analyze- It, EP Evaluator) Statistical calculators, graph paper Data- Crunching Tools

24  One quantitative test taken through the validation process  One qualitative method taken through the validation process How We Will Work Through This Module 24

25 Reportable Range Precision Accuracy Reference Intervals Sensitivity Specificity Elements of Validation 25

26  Repeat testing over short and long term (one day and 20 days, respectively)  20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)  Standard solutions  Control materials  Pools (short term only) Precision  Definition: Reproducibility  Gives information related to random error Introduction What is needed How we perform the testing 26

27 Precision: How We Evaluate the Data 27  Mean  Standard deviation (SD)  Coefficient of Variation (CV)  Short term: 0.25 of allowable total error  Long term: 0.33 of allowable total error Calculate the following: What amount of random error is allowable, based on CLIA criteria?

28 Link for:  Clinical Laboratory Improvement Amendments (CLIA)  College of American Pathologists (CAP)  Royal College of Pathologists of Australasia (RCPA)  Others Allowable Total Error Database 28

29 Precision: Levey-Jennings (LJ) Charts 29 Values Run

30 Precision: How We Evaluate the Data 30  Mean  SD  CV: More commonly used, allows for easier comparison How do we compare to manufacturer’s data?

31 Precision Example 31 Mean of Level 1 Glucose CLIA Total Allowable Error Total Allowable Error Level 1 Glucose Random error allowed: 90 mg/dL 6 mg/dL or ± 10% 0.1 x 90 = 9 mg/dL 0.25 x total allowable 0.25 x 9 mg/dL 2.25 mg/dL 0.33 x total allowable 0.33 x 9 mg/dL 2.97 mg/dL Long-term precision Short-term precision

32  Work with Levey-Jennings graph and data  Work with mean and standard deviation to calculate a coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation  Determine if precision data is acceptable Activity 32

33 Accuracy  Definition: How close to the true value  Comparison of methods  Gives information related to systematic error  Potential conflicts on interpretation of results (reference values) Introduction  40 different specimens  Cover reportable range of method  Quality versus quantity What is needed  Duplicate measurements of each specimen on each method  Minimum of five days, prefer over 20 (since replicate testing is same) How we perform the testing 33

34 Accuracy: How We Evaluate the Data Graph the Data: Test method on Y- axis Reference (comparative) method on X-axis Shows analytical range of data, linearity of response over range and relationship between methods Real time Difference plot Comparison plot Calculate y = mx + b b represents constant error m represents proportional error 34

35 Visual Inspection for Accuracy 35 Test Method Reference Method Intercept (x 1, y 1 ) (x 2, y 2 ) Slope = (y 2 - y 1 ) / (x 2 - x 1 )

36  Slope: Usually not significantly different from 1  Intercept: Not significantly different from 0  Significant difference with Medical Decision Points Accuracy: How We Evaluate the Data 36

37 Slope  Measure of proportional bias  m = (y1-y2)/(x1-x2) or “rise/run”  Slope greater than 1 means the Y (Test) values are generally higher than the X (Comparative) values  Slope of 1.11 means the Y (Test) values are on average 11% higher than the X (Comparative) values Calculate Appropriate Statistics 37

38 Intercept of the Line  Measure of constant bias between two methods  Y (Test) value at the point where the line crosses the Y axis  If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values Calculate Appropriate Statistics (cont'd) 38

39 Accuracy 39 What type of bias do you see?

40 Accuracy (cont’d) Constant BiasProportional Bias 40

41 Can a linear regression formula offer predictive value in relation to method comparisons? A.Yes B.No Skill Check 41

42  Create graph based on sample set  Determine slope from best-fit line  Determine Y-intercept from best-fit line  Explain the relationship between comparative and test results Activity 42

43  CLSI recommends four measurements of each specimen; three are sufficient  Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)  Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens  At least four levels (five preferred) Reportable Range / Linearity  Definition: Lowest and highest test results that are reliable  Especially important with two point calibrations  Analytical Measurement Range (AMR) and derived Clinical Reportable Range (CRR) Introduction What is needed How we perform the testing 43

44 Reportable Range: How We Evaluate the Data 44  Measured values on Y-axis versus  Known or assigned values on X-axis Plot mean values of: Compare with expected values (typically provided by manufacturer) Visually inspect, draw best-fit line, estimate reportable range

45 Reportable Range Activity Assigned Value Experimental Results AverageRep #1Rep #2Rep #3Rep #4 10.0____ ____ ____ ____ ____

46 Reportable Range Activity (cont'd) Assigned Value Experimental Results AverageRep #1Rep #2Rep #3Rep #

47 Reportable Range Activity (cont'd) 47

48 AMR vs. CRR Analytical Measurement Range (AMR) Linearity Clinically Reportable Range (CRR) Allows for dilution or other preparatory steps beyond routine 48

49 If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely? A.AMR B.CRR C.Neither A or B D.Both A and B Skill Check 49

50 Utilizing the marketing materials from the two chemistry linearity kits in your handouts: 1.Determine which kit would be more appropriate for use with the chemistry assay you chose earlier 2.Explain your reasoning Linearity Materials 50

51 Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide. 1.Review data; record any initial observations 2.Graph data on supplied graph paper 3.Determine your assay’s AMR Graph Activity 51

52 LevelRep 1Rep 2Rep 3Rep Linearity Experiment Results 52

53 Activity Using an Excel spreadsheet, create a graph and calculate linear regression statistics from the data provided 53

54 Rep 1Rep 2Rep 3Rep 4 Lab's Average Known Conc

55 55

56  Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is 15 – 1400 mg/dL  Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP Dilution Protocols 56

57 Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results? 1.12 mg/dL mg/dL mg/dL Reportable Results 57

58 Reference Intervals 58  Definition: Normal range in healthy population  Used for diagnosis/clinical interpretation of results Introduction  Pre-defined “normal” criteria for screening purposes  Transferring: 20 “normal” individuals’ specimens  Establishing: 120 “normal” individuals’ specimens What is needed  Perform testing on all samples  Document results How we perform the testing

59 TransferringEstablishing  18 of 20 must fall within manufacturer’s ranges  Calculate mean and SD of data for each group  Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended) 59 Reference Intervals: How We Evaluate the Data

60 Activity 60  Determine if assay is eligible for transference of reference intervals  Review a sample set of data to determine if transference may be performed; if not, determine next step(s)

61 Sensitivity  Definition: Lowest reliable value; lower limit of detection, especially of interest in drug testing and tumor markers  Different terminologies used by different manufacturers Introduction  Blank solutions  Spiked samples What is needed  20 replicate measurements over short or long term, depending on focus How we perform the testing 61

62 Sensitivity: How We Evaluate the Data Lower Limit of Detection (LLD): Mean of the blank sample, plus two or three SD of blank sample Biological Limit of Detection: LLD plus two or three times SD of spiked sample with concentration of detection limit Biological Limit of Detection: LLD plus two or three times SD of spiked sample with concentration of detection limit Functional Sensitivity: Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is reliable Functional Sensitivity: Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is reliable Three methods used: 62

63 Activity 63 Using the manufacturer’s package inserts, find the related information for sensitivity. How was it calculated?

64 Specificity  Definition: Determination of how well a method measures the analyte of interest accompanied by potential interfering materials Introduction  Standard solutions, participant specimens or pools  Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations What is needed  Duplicate measurements How we perform the testing 64

65 Specificity: How We Evaluate the Data  Tabulate results for pairs of samples (dilution and interferent)  Calculate means for each (dilution and interferent)  Calculate the differences  Calculate the average interference of all specimens tested at a given concentration of interference 65

66  Compare diagnosis  Assume comparative (reference) method is accurate  Determine the following:  True Positives, True negatives  False Positives, False negatives  Calculate sensitivity and specificity and compare to manufacturer Qualitative Assays 66

67  Negative and Positive Quality Controls  Use QC materials recommended by manufacturer for verification purposes  Determine validity of other results, e.g., method comparisons  Evaluate failed runs if they occur during verification process Qualitative Assays: Control of Validation 67

68  How is it performed?  Runs of specimens with analyte concentrations near the cutoff point  Three specimens, one at cutoff, one just below cutoff, and one just above cutoff (± 20% recommended)  Replicate measurements of each of three specimens (20 each, minimum)  How is it evaluated?  Determine percentage of positives and negatives for each specimen  Evaluate cutoff, as well as other two specimens Qualitative Methods: Precision 68

69  How is it performed?  Specimens typical of population (to be tested in future use of method)  50 positive specimens and 50 negative specimens recommended; minimum 20 each  Performed over 10 to 20 days  How is it evaluated?  Discrepant results near cutoff?  Most often sensitivity and specificity used to describe performance Accuracy/Method Comparisons 69

70 Qualitative Methods Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 70  False Positive Rate - False Positives divided by total number of Negatives  False Negative Rate - False Negatives divided by total number of Positives True vs. False

71 Qualitative Methods (cont'd) Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 71  Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)  Specificity = 100 x True Negatives divided by (True Negatives + False Positives)

72 Qualitative Methods (cont'd) Comparative or Reference Method Result PositiveNegative Test Method Result PositiveTrue PositiveFalse Positive Positive Predictive Value NegativeFalse NegativeTrue Negative Negative Predictive Value SensitivitySpecificity 72  Predictive Values - Operation of a test on a mixed population of Positive and Negatives  A property of the test and the population; and affected by prevalence of Positives  Positive Predictive Value = True Positives divided by (True Positives + False Positives)  Negative Predictive Value = True Negatives divided by (True Negatives + False Negatives)

73  High Diagnostic Value  100% Sensitivity  100% Specificity  What happens if True Positive rate is equal to the False Positive rate? Evaluation Criteria 73

74 Activity 74 Estimate sensitivity and specificity of a qualitative method given a data set.

75 Activity (cont’d) 75 Create a validation plan for a quantitative assay to be performed in your laboratory.

76 Now that you have completed this module, you should be able to:  Identify test classifications  Define what each validation experiment details for testing methods  Discuss what is recommended to perform each of the validation experiments for testing methods  Recognize how to evaluate data obtained from each of the validation experiments In Closing 76

77 A rapid HIV test would likely be classified as a: A.High complexity, modified assay B.Moderate complexity, unmodified assay C.FDA-approved, modified assay D.Waived, FDA-approved, unmodified assay Post-Assessment Question #1 77

78 The precision of a test method gives information related to the method’s: A.Systematic error B.Comparison of results to a reference method C.Reproducibility D.Likelihood of being affected by hemolysis, lipemia and icterus E.Both A and B Post-Assessment Question #2 78

79 When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals? A.20 B.18 C.16 D.15 Post-Assessment Question #3 79

80 Which linear regression equation component gives information regarding constant bias? A.y B.x C.m (slope) D.b (intercept) Post-Assessment Question #4 80

81  DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines.   Validation of Qualitative Methods.  42 CFR §  College of American Pathologists Commission on Laboratory Accreditation, Accreditation Checklists, April  Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard QC, Inc.,  Clinical and Laboratory Standards Institute. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA,  Clinical and Laboratory Standards Institute. Evaluation of Precision.  Performance of Quantitative Measurement Methods. NCCLS document EP5-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA,  Clinical and Laboratory Standards Institute. User verification of Performance for Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, References 81

82 Wrap Up 82


Download ppt "This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases,"

Similar presentations


Ads by Google