Presentation on theme: "R Method Validation Studies Its a Jungle out There!"— Presentation transcript:
r Method Validation Studies Its a Jungle out There!
r Well Duh...
r Purposes of Method Validation Studies: To quantifiably characterize system performance To assess potential for error To identify method-to-method differences To meet regulatory guidelines
r When to Do Method Validation Studies: When considering purchasing a new system When placing a new system into service At regular intervals to assess on-going system performance When troubleshooting questionable system performance
r Basic Components of Method Validation Studies: Any or All of the Following... Precision Studies Accuracy Studies Range Validation/Verification Studies Reportable Range Critical Limits Analytical Measurement Range Clinically Reportable Range Calibration Verification / Linearity Reference Range Studies
r How to Do Method Validation Studies: Dont wing it!!! Consult appropriate reference guides. NCCLS evaluation document series Manufacturers package inserts, evaluation protocol manuals and supplemental protocols.
r Precision Studies Characterize the reproducibility of a test system Performed by repetitive testing of the same sample Intra-run Inter-run Inter-instrument Statistical parameters Mean Standard Deviation (SD), mg/dL Coefficient of Variation (CV), % Acceptability Expectations…the lower the better
r Accuracy Studies Method-to-method studies comparing a POC instrument to a properly calibrated primary Laboratory analyzer. For the purpose of assessing accuracy, the same samples should be tested on both the Laboratory analyzer and the Point-of-Care instrument.
r Accuracy Studies, contd: A typical Accuracy Study includes: all, or a representative sub-set, of Point-of-Care instruments optimally at least 40 different samples (minimum of 20 for statistical validity) good distribution of sample values ranging from low to high same samples tested on Point-of-Care instruments and Laboratory analyzer. If capillary samples are compared to venous glucose results, use only samples from fasting patients (Capillary glucose levels can be mg/dL higher than venous levels at the same time in the non-fasting state).
r Accuracy Studies, contd Statistical analysis of accuracy study results typically includes: Regression analysis Bias analysis
r Accuracy Studies, Contd
r Regression Analysis Plotting the Comparative Data...
r Accuracy Studies, contd Regression Statistics Review: Correlation Coefficient (r) - characterizes the dispersion of results around the line of best fit. Slope - The lean of the line of best fit (proportional bias) Y-Intercept - the point at which the line of best fit intersects the Y axis. (constant bias) Acceptability Criteria: Correlation Coefficient (r) - the closer to 1.0 the better Slope - The closer to 1.0 the better Y-Intercept - the closer to zero the better
r Accuracy Studies, Contd The Equation of the Line of Best Fit: y = mx + b m = Slope b = Y Intercept
r Accuracy Studies, Contd The Equation of the Line of Best Fit y = 0.991x Slope Y-intercept
r Accuracy Studies, Contd The Equation of the Line of Best Fit y = 0.991x Slope Y-intercept
r Accuracy Studies, contd: Bias Analysis: Calculate mg/dL or % difference between Point-of- Care instrument and Laboratory analyzer for each set of paired data-points. Acceptability Criteria (Glucose) : 95% of meter results within +/- 15 mg/dL of Laboratory analyzer results below 75 mg/dL or within +/-20% of Laboratory analyzer results equal to or greater than 75 mg/dL (NCCLS document C30A-2)
r Bias Analysis, Contd Bias Chart
r Range Validation Studies Definitions Reportable Range - The range of results for which a system has been proven to yield numerically accurate results Analytical Measurement Range (AMR) - The range of results through which a method yields numerical values Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols Calibration Verification - The process of verifying that a system is properly calibrated Critical Limits - Low and high result limits which, when exceeded, require follow up action Reference Range - Normal range
r Range Validation Studies Definitions Analytical Measurement Range (AMR) - The range of results through which a method yields numerical values Clinically reportable range - (CRR) An extension of the AMR supplemented by dilution or concentration protocols Calibration Verification - The process of verifying that a system is properly calibrated Reportable Range - The range of results for which a system has been proven to yield numerically accurate results Critical Limits - Low and high result limits which, when exceeded, require follow up action Reference Range - Normal range
r Range Validation Studies, Contd Validation of Reportable Range, Analytical Measurement Range, Reference Range, Critical Limits and Calibration Verification (CAP POC July 2003 Checklist, Page 34 Commentary) Can all be accomplished through the same protocol if the range of testing spans the broadest range (AMR) Materials used must be matrix appropriate such as: Split blood samples linearity/calibration verification materials supplied by manufacturer or other vendor However...
r CAP POC If the laboratory has more than one method-system for performing tests for a given analyte, are they checked against each other at least twice a year for correlation of patient results? NOTE: This includes same or different instrument makes/models. This comparison must include all instruments. If multiple methods are used for a particular analyte, comparing a subset of each method to one another is acceptable only after all like instruments have been compared to one another to ensure correlation and establish that a representative subset is used in the method-to-method correlation.
r CAP POC.08700, Contd: Significant changes to the commentary!!! POCT glucose devices are a special case in which a large number of devices may be in use in an institution. In this case alternate approaches are acceptable for documenting correlation of patient results among the devices and with the main laboratory. One approach is to document for a lot of reagent strips the agreement between patient results for a laboratory method and representative POCT device(s) and simultaneously collect results for QC materials run on the POCT devices. Other correlation schemes may be suitable. The critical factor is to have a mechanism that documents the agreement between patient results performed by the POCT glucose devices and the main laboratory.
r Where to Find the Method Validation Regulations CLIA CAP JCAHO
r Regulatory Guidelines for Method Validation: CLIA 88 Final Rule - (January 24, 2003) CAP General Checklist (July 2003) CAP Point-of-Care Checklist (July 2003) JCAHO - WT and QC Section (2004 Pre-Publication with Crosswalks from 2003 Guidelines) NCCLS - C30A-2 - (Item 6.3.2).
r CLIA Guidelines for Method Validation: CLIA 88 Final Rule - Subpart K (Quality Systems for Non-waived Tests) Revised January 24, 2003 Outlines requirements for unmodified non-waived, FDA cleared tests By definition, waived tests are not subject to items in Subpart K
r CAP Guidelines for Method Validation: CAP Laboratory General Checklist Revised July 2003 (GEN GEN.42163) Does not distinguish based on CLIA designated test complexity Requires validation of precision (intra- and inter-run) and accuracy Requires verification of analytic sensitivity (Low end) Requires assessment of analytical interference Requires verification of reportable range (AMR) Requires verification of reference range
r CAP Guidelines for Method Validation: CAP Laboratory Point-of-Care Checklist Revised July 2003 (POC POC.08700) Does not distinguish based on CLIA designated test complexity Requires calibration verification Requires validation of Analytical Measurement Range Requires method-to-method and instrument-to- instrument correlation
r JCAHO Guidelines for Method Validation: WT Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) Item WT Non-specific reference to requiring Equipment performance evaluation QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) Items QC Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests. Requires reference range verification for all new methods
r JCAHO Guidelines for Method Validation: QC Section (2004 Pre-Publication with Crosswalk from 2003 Guidelines) Item QC.1.70 Method Validation requirements for precision, accuracy and reportable range apply to FDA cleared, unmodified moderate complexity tests. Requires reference range verification for all new methods Item QC.1.80 Method-to-method and instrument-to-instrument requirement specifies that waived tests are exempt.
r Informative Regulatory Links (CLIA Regs with 1/24/03 updates) (Oct. 29th Teleconference y+services/standards/new+standards/2004+standards.h tm (JCAHO Pre-publication 2004 Standards with Crosswalk) (CAP Checklist download page) (EP Evaluator Release 5)
r Links to Informative Articles ge_barrier.html (CAP Today on AMR/CRR and Calibration Verification) (CLIA 1/24/03 method validation changes by Sharon Ehrmyeyer, Ph.D) (CLIA Key 1/24/03 Changes by Dr Westgard) (Method Validations: The Inner, Hidden, Deeper, Secret Meaning by Dr Westgard)
r Method Validation Studies Questions and Discussion