Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003

Outline Introduction PCR based quantitative nucleic acid assays Literature references Acceptance criteria Examples References

Methods Comparison Studies: to validate a new assay Purposes: To show that the new assay has good agreement with the reference assays To show that the assay performs similarly with different types of specimen Premises of methods comparison studies: A linear relationship between the two assays LOD, dynamic range have to be already established Appropriate transformation to normalize the data Analysis: To detect constant bias and proportional bias

Constant Bias: the difference between the two methods is constant across the data range

Proportional Bias: the difference between the two methods is linear across the data range

PCR based nucleic acid assays To quantify the viral load by PCR method Characteristics: A wide dynamic range (e.g. 10cp/mL to 1E7 cp/mL) Skewed distribution (non-normal): typically log10 transformation for the data Heteroscedasticity: variance is higher at higher titer levels log10 transformation may not achieve homogeneity in variance (variance at lower end may increase) Other transformation:

PCR based assays: a wide dynamic range - data are log10 transformed

PCR based assays: log10 transformation may remove some skewness Untransformed log10 transformed

Literature references on Methods Comparison Studies Correlation coefficient Other coefficients T-test Bland-Altman plot Ordinary least squares regression Passing-Bablok regression Deming regression

Correlation coefficient R or R2 Measures the strength of linear relationship between two assays Does not measure agreement: cannot detect constant or proportional bias Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

Other coefficients Concordance coefficient (Lin, 1989): Measures the strength of relationship between two assays that fall on the 45o line through the origin Gold-standard correlation coefficient (St.Laurent 1998): Measures the agreement between a new assay and a gold standard

T-test Paired t-test on the difference in the measurements by two assays Can only detect constant bias Cannot detect proportional bias

Bland-Altman graphical analysis (Bland and Altman, 1986) Methods: Plot the Difference of the two assays (D = X-Y) vs. the Average of the two assays (A = (X+Y)/2) Visually inspect the plot and see if there are any trends in the plot  proportional bias Summarize the bias between the two assays by the mean, SD, 95% CI  constant bias Modification: regress D with A, test if slope = 0 (Hawkins, 2002) A useful visual tool: transformation, heteroscedasticity, outliers, curvature

Bland Altman plot (continued)

Ordinary least-squares regression Methods: Regress the observed data of the new assay (Y) with those of the reference assay (X) Minimize the squared deviations from the identity line in the vertical direction Modifications: weighted least squares Assumptions: The reference assay (X) is error free, or the error is relatively small compared to the range of the measurements e.g. in clinical chemistry studies, the measurement errors are minimal

Ordinary least-squares regression (continued) If measurement errors exist in both assays, the estimates are biased slope tends to be smaller intercept tends to be larger

Passing-Bablok regression (Passing and Bablok, 1983) A nonparametric approach - robust to outliers Methods: Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate) Confidence intervals by the rank techniques Assumptions: The measurement errors in both assays follow the same type of distribution (not necessarily normal) The ratio of the variance is a constant (variance not necessarily constant across the range of data) The sampling distributions of the samples are arbitrary

Deming regression (Linnet, 1990) Methods: Orthogonal least squares estimates: minimize the squared deviation of the observed data from the regression line Standard errors for the estimates obtained by Jackknife method Weighted Deming regression when heteroscedastic Assumptions: Measurement errors for both assays follow independent normal distributions with mean 0 Error variances are assumed to be proportional (variance not necessarily constant across the range of data)

Comparison of the 3 regression methods (Linnet, 1993) Electrolyte study (homogeneous variance): OLS, Passing-Bablok: biased slope, large Type I error, larger RMSE than Deming Deming: unbiased slope, correct Type I error Metabolite study (heterogeneous variance): All have unbiased slope estimates Weighted LS and weighted Deming are most efficient Type I error is large for OLS, weighted LS, Deming and Passing-Bablok Presence of outliers: Passing-Bablok is robust to outliers Deming regression requires detection of outliers

Software Statistical packages: SAS, Splus Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression): Analyse-it (Excel add-on): does not support weighted Deming regression Method Validator (a freeware) CBStat (Linnet K.)

Acceptance criteria for regression type analysis Independent acceptance criteria for slope and intercept estimates: e.g. slope estimate within (0.9, 1.1), intercept estimate within (-0.2, 0.2) Drawback: asymmetrical acceptance region across the data range

Asymmetrical acceptance region

Proposed acceptance criteria Goals: to show that the new assay is ‘equivalent’ to the reference assay to demonstrate that the bias between the two assays is within some acceptable threshold across the clinical range Acceptance Criteria: Choice of tolerance level A: accuracy specification for the new assay

Mathematical models

Comparison of the acceptance criteria: {Int  (-0. 2,0. 2), Slope  (0 Comparison of the acceptance criteria: {Int  (-0.2,0.2), Slope  (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

Acceptance region for the parameters: criteria for the intercept and slope are dependent

where A is the accuracy specification of the new assay Equivalence test where A is the accuracy specification of the new assay Methods: If the 90% two-sided confidence interval of the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent Deming-Jackknife is used to do the estimation

where  = Var()/Var() (assumed known or to be estimated) Deming regression: (a.k.a. errors-in-variables regression, a structural or functional relationship model) Minimize the sum of squares: where  = Var()/Var() (assumed known or to be estimated) The solutions are given by: Weighted Deming regression:

Estimation of  in Deming regression Duplicate measurements: >2 replicates: residual errors by ANOVA Mis-specification of  (Linnet 1998): biased slope estimate large Type I error

Jackknife estimation: to obtain the final parameter estimates and the SEs Omit one pair of data at a time, obtain the Deming-regression estimates: The ith pseudo-values of the intercept and slope are: Final estimates and SEs for  and  are the mean and standard error of i and i

Bias estimation by Jackknife At each nominal level , the ith pseudo-value of the Bias is: The bias estimate and the SE at each nominal level are the mean and SE of Biasi The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A) The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A)

Example 1: methods comparison for two HIV-1 assays

Bland-Altman plot: potential outliers in the data

Identify outliers: fitting a linear regression line to the Bland Altman plot

Remove outliers: Bland-Altman plot shows no trend in Difference vs Remove outliers: Bland-Altman plot shows no trend in Difference vs. Average mean difference = 0.02 (95% CI: -0.06, 0.10) slope = 0.033 (p-value = 0.5)

Regression analysis: results from the 3 methods are very similar

Bias estimation: almost all 90% CI lie within the tolerance bounds (-0

Example 2: to show matrix equivalency between EDTA Plasma and Serum

mean difference = -0.06 (95% CI: -0.16, 0.04) Bland-Altman plot on average titer: most titers higher than 1E5 IU/mL, heteroscedasticity? slope = 0.03 (p-value = 0.6) mean difference = -0.06 (95% CI: -0.16, 0.04)

Checking for heteroscedasticity: residual errors from random effects models

  1: Pooled within-sample SD for EDTA = 0   1: Pooled within-sample SD for EDTA = 0.0706 Pooled within-sample SD for Serum = 0.0715

Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

References Bland M., Altman D. (1986). ‘Statistical methods for assessing agreement between two methods of clinical measurement’. Lancet 347: 307-310. Hawkins D. (2002). ‘Diagnostics for conformity of paired quantitative measurements’. Stat in Med 21: 1913-1935. Lin L.K. (1989). ‘A concordance correlation coefficient to evaluate reproducibility’. Biometrics 45: 255-268. Linnet K. (1990). ‘Estimation of the linear relationship between the measurements of two methods with proportional bias’. Stat in Med 9: 1463-1473. Linnet K. (1993). ‘Evaluation of regression procedures for methods comparison studies’. Clin Chem 39: 424-432. Linnet K. (1998). ‘Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies’. Clin Chem 44: 1024-1031. Linnet K. (1999). ‘Necessary sample size for method comparison studies based on regression analysis’. Clin Chem 45: 882-894.

References (continued) Passing H., Bablok W. (1983). ‘A new biometrical procedure for testing the equality of measurements from two different analytical methods’. J Clin Chem Clin Biochem 21: 709-720. Passing H., Bablok W. (1984). ‘Comparison of several regression procedures for method comparison studies and determination of sample sizes’. J Clin Chem Clin Biochem 22: 431-445. St. Laurent R.T. (1998). ‘Evaluating Agreement with a Gold Standard in Method Comparison Studies’. Biometrics 54: 537-545.