3 Purposes of Method Validation Studies: To quantifiably characterize system performanceTo assess potential for errorTo identify method-to-method differencesTo meet regulatory guidelines
4 When to Do Method Validation Studies: When considering purchasing a new systemWhen placing a new system into serviceAt regular intervals to assess on-going system performanceWhen troubleshooting questionable system performance
5 Basic Components of Method Validation Studies: Any or All of the Following... Precision StudiesAccuracy StudiesRange Validation/Verification StudiesReportable RangeCritical LimitsAnalytical Measurement RangeClinically Reportable RangeCalibration Verification / LinearityReference Range Studies
6 How to Do Method Validation Studies: Don’t wing it How to Do Method Validation Studies: Don’t wing it!!! Consult appropriate reference guides.NCCLS evaluation document seriesManufacturer’s package inserts, evaluation protocol manuals and supplemental protocols.
7 Precision Studies Characterize the reproducibility of a test system Performed by repetitive testing of the same sampleIntra-runInter-runInter-instrumentStatistical parametersMeanStandard Deviation (SD), mg/dLCoefficient of Variation (CV), %Acceptability Expectations…the lower the better
8 Accuracy StudiesMethod-to-method studies comparing a POC instrument to a properly calibrated primary Laboratory analyzer.For the purpose of assessing accuracy, the same samples should be tested on both the Laboratory analyzer and the Point-of-Care instrument.
9 Accuracy Studies, cont’d: A typical Accuracy Study includes:all, or a representative sub-set, of Point-of-Care instrumentsoptimally at least 40 different samples (minimum of 20 for statistical validity)good distribution of sample values ranging from low to highsame samples tested on Point-of-Care instruments and Laboratory analyzer. If capillary samples are compared to venous glucose results, use only samples from fasting patients (Capillary glucose levels can be mg/dL higher than venous levels at the same time in the non-fasting state).
10 Accuracy Studies, cont’d Statistical analysis of accuracy study results typically includes:Regression analysisBias analysis
12 Regression Analysis Plotting the Comparative Data...
13 Regression Analysis Plotting the Comparative Data...
14 Regression Analysis Plotting the Comparative Data...
15 Accuracy Studies, cont’d Regression Statistics Review:Correlation Coefficient (r) - characterizes the dispersion of results around the line of best fit.Slope - The “lean” of the line of best fit (proportional bias)Y-Intercept - the point at which the line of best fit intersects the Y axis. (constant bias)Acceptability Criteria:Correlation Coefficient (r) - the closer to 1.0 the betterSlope - The closer to 1.0 the betterY-Intercept - the closer to zero the better
16 Accuracy Studies, Cont’d The Equation of the Line of Best Fit:y = mx + bm = Slopeb = Y Intercept
17 Accuracy Studies, Cont’d The Equation of the Line of Best Fit y = 0.991x + 8.4SlopeY-intercept
18 Accuracy Studies, Cont’d The Equation of the Line of Best Fit y = 0.991x + 8.4SlopeY-intercept
19 Accuracy Studies, cont’d: Bias Analysis:Calculate mg/dL or % difference between Point-of-Care instrument and Laboratory analyzer for each set of paired data-points.Acceptability Criteria (Glucose):95% of meter results within +/- 15 mg/dL of Laboratory analyzer results below 75 mg/dL or within +/-20% of Laboratory analyzer results equal to or greater than 75 mg/dL (NCCLS document C30A-2)
21 Range Validation Studies DefinitionsReportable Range - The range of results for which a system has been proven to yield numerically accurate resultsAnalytical Measurement Range (AMR) - The range of results through which a method yields numerical valuesClinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocolsCalibration Verification - The process of verifying that a system is properly calibratedCritical Limits - Low and high result limits which, when exceeded, require follow up actionReference Range - Normal range
22 Range Validation Studies DefinitionsAnalytical Measurement Range (AMR) - The range of results through which a method yields numerical valuesClinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocolsCalibration Verification - The process of verifying that a system is properly calibratedReportable Range - The range of results for which a system has been proven to yield numerically accurate resultsCritical Limits - Low and high result limits which, when exceeded, require follow up actionReference Range - Normal range
23 Range Validation Studies, Cont’d Validation of Reportable Range, Analytical Measurement Range, Reference Range, Critical Limits and Calibration Verification (CAP POC July 2003 Checklist, Page 34 Commentary)Can all be accomplished through the same protocol if the range of testing spans the broadest range (AMR)Materials used must be “matrix appropriate” such as:Split blood sampleslinearity/calibration verification materials supplied by manufacturer or other vendorHowever...
24 CAP POC If the laboratory has more than one method-system for performing tests for a given analyte, are they checked against each other at least twice a year for correlation of patient results? NOTE: This includes same or different instrument makes/models. This comparison must include all instruments. If multiple methods are used for a particular analyte, comparing a subset of each method to one another is acceptable only after all like instruments have been compared to one another to ensure correlation and establish that a representative subset is used in the method-to-method correlation.
25 CAP POC.08700, Cont’d: Significant changes to the commentary!!! “POCT glucose devices are a special case in which a large number of devices may be in use in an institution. In this case alternate approaches are acceptable for documenting correlation of patient results among the devices and with the main laboratory. One approach is to document for a lot of reagent strips the agreement between patient results for a laboratory method and representative POCT device(s) and simultaneously collect results for QC materials run on the POCT devices. ““Other correlation schemes may be suitable. The critical factor is to have a mechanism that documents the agreement between patient results performed by the POCT glucose devices and the main laboratory.”
26 Where to Find the Method Validation Regulations CLIACAPJCAHO
27 Regulatory Guidelines for Method Validation: CLIA 88 Final Rule - (January 24, 2003)CAP General Checklist (July 2003)CAP Point-of-Care Checklist (July 2003)JCAHO - WT and QC Section (2004 Pre-Publication with Crosswalks from 2003 Guidelines)NCCLS - C30A-2 - (Item 6.3.2).
28 CLIA Guidelines for Method Validation: CLIA 88 Final Rule - Subpart K (Quality Systems for Non-waived Tests)Revised January 24, 2003Outlines requirements for unmodified “non-waived”, FDA cleared testsBy definition, waived tests are not subject to items in Subpart K
29 CAP Guidelines for Method Validation: CAP Laboratory General ChecklistRevised July 2003 (GEN GEN.42163)Does not distinguish based on CLIA designated test complexityRequires validation of precision (intra- and inter-run) and accuracyRequires verification of analytic sensitivity (Low end)Requires assessment of analytical interferenceRequires verification of reportable range (AMR)Requires verification of reference range
30 CAP Guidelines for Method Validation: CAP Laboratory Point-of-Care ChecklistRevised July 2003 (POC POC.08700)Does not distinguish based on CLIA designated test complexityRequires calibration verificationRequires validation of Analytical Measurement RangeRequires method-to-method and instrument-to-instrument correlation
31 JCAHO Guidelines for Method Validation: WT Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)Item WT Non-specific reference to requiring “Equipment performance evaluation”QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)Items QC Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests.Requires reference range verification for all new methods
32 JCAHO Guidelines for Method Validation: QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines)Item QC.1.70Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests.Requires reference range verification for all new methodsItem QC.1.80Method-to-method and instrument-to-instrument requirement specifies that waived tests are exempt.
34 Links to Informative Articles (CAP Today on AMR/CRR and Calibration Verification)(CLIA 1/24/03 method validation changes by Sharon Ehrmyeyer, Ph.D)(CLIA Key 1/24/03 Changes by Dr Westgard)(Method Validations: The Inner, Hidden, Deeper, Secret Meaning by Dr Westgard)
35 Method Validation Studies Questions and Discussion