1. Methods Comparison Studies for Quantitative Nucleic Acid Assays
Jacqueline Law, Art DeVault
Roche Molecular Systems
Sept 19, 2003

2. Outline Introduction PCR based quantitative nucleic acid assays
Literature references
Acceptance criteria
Examples
References

3. Methods Comparison Studies: to validate a new assay
Purposes:
To show that the new assay has good agreement with the reference assays
To show that the assay performs similarly with different types of specimen
Premises of methods comparison studies:
A linear relationship between the two assays
LOD, dynamic range have to be already established
Appropriate transformation to normalize the data
Analysis:
To detect constant bias and proportional bias

4. Constant Bias: the difference between the two methods is constant across the data range

5. Proportional Bias: the difference between the two methods is linear across the data range

6. PCR based nucleic acid assays
To quantify the viral load by PCR method
Characteristics:
A wide dynamic range (e.g. 10cp/mL to 1E7 cp/mL)
Skewed distribution (non-normal): typically log10 transformation for the data
Heteroscedasticity: variance is higher at higher titer levels
log10 transformation may not achieve homogeneity in variance (variance at lower end may increase)
Other transformation:

7. PCR based assays: a wide dynamic range - data are log10 transformed

8. PCR based assays: log10 transformation may remove some skewness
Untransformed
log10 transformed

9. Literature references on Methods Comparison Studies
Correlation coefficient
Other coefficients
T-test
Bland-Altman plot
Ordinary least squares regression
Passing-Bablok regression
Deming regression

10. Correlation coefficient R or R2
Measures the strength of linear relationship between two assays
Does not measure agreement: cannot detect constant or proportional bias
Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

11. Other coefficients Concordance coefficient (Lin, 1989):
Measures the strength of relationship between two assays that fall on the 45o line through the origin
Gold-standard correlation coefficient (St.Laurent 1998):
Measures the agreement between a new assay and a gold standard

12. T-test
Paired t-test on the difference in the measurements by two assays
Can only detect constant bias
Cannot detect proportional bias

13. Bland-Altman graphical analysis (Bland and Altman, 1986)
Methods:
Plot the Difference of the two assays (D = X-Y) vs. the Average of the two assays (A = (X+Y)/2)
Visually inspect the plot and see if there are any trends in the plot  proportional bias
Summarize the bias between the two assays by the mean, SD, 95% CI  constant bias
Modification: regress D with A, test if slope = 0 (Hawkins, 2002)
A useful visual tool:
transformation, heteroscedasticity, outliers, curvature

14. Bland Altman plot (continued)

15. Ordinary least-squares regression
Methods:
Regress the observed data of the new assay (Y) with those of the reference assay (X)
Minimize the squared deviations from the identity line in the vertical direction
Modifications: weighted least squares
Assumptions:
The reference assay (X) is error free, or the error is relatively small compared to the range of the measurements
e.g. in clinical chemistry studies, the measurement errors are minimal

16. Ordinary least-squares regression (continued)
If measurement errors exist in both assays, the estimates are biased
slope tends to be smaller
intercept tends to be larger

17. Passing-Bablok regression (Passing and Bablok, 1983)
A nonparametric approach - robust to outliers
Methods:
Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate)
Confidence intervals by the rank techniques
Assumptions:
The measurement errors in both assays follow the same type of distribution (not necessarily normal)
The ratio of the variance is a constant (variance not necessarily constant across the range of data)
The sampling distributions of the samples are arbitrary

18. Deming regression (Linnet, 1990)
Methods:
Orthogonal least squares estimates: minimize the squared deviation of the observed data from the regression line
Standard errors for the estimates obtained by Jackknife method
Weighted Deming regression when heteroscedastic
Assumptions:
Measurement errors for both assays follow independent normal distributions with mean 0
Error variances are assumed to be proportional (variance not necessarily constant across the range of data)

19. Comparison of the 3 regression methods (Linnet, 1993)
Electrolyte study (homogeneous variance):
OLS, Passing-Bablok: biased slope, large Type I error, larger RMSE than Deming
Deming: unbiased slope, correct Type I error
Metabolite study (heterogeneous variance):
All have unbiased slope estimates
Weighted LS and weighted Deming are most efficient
Type I error is large for OLS, weighted LS, Deming and Passing-Bablok
Presence of outliers:
Passing-Bablok is robust to outliers
Deming regression requires detection of outliers

20. Software Statistical packages: SAS, Splus
Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression):
Analyse-it (Excel add-on): does not support weighted Deming regression
Method Validator (a freeware)
CBStat (Linnet K.)

21. Acceptance criteria for regression type analysis
Independent acceptance criteria for slope and intercept estimates:
e.g. slope estimate within (0.9, 1.1), intercept estimate within (-0.2, 0.2)
Drawback: asymmetrical acceptance region across the data range

22. Asymmetrical acceptance region

23. Proposed acceptance criteria
Goals:
to show that the new assay is ‘equivalent’ to the reference assay
to demonstrate that the bias between the two assays is within some acceptable threshold across the clinical range
Acceptance Criteria:
Choice of tolerance level A:
accuracy specification for the new assay

24. Mathematical models

25. Comparison of the acceptance criteria: {Int  (-0. 2,0. 2), Slope  (0
Comparison of the acceptance criteria: {Int  (-0.2,0.2), Slope  (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

26. Acceptance region for the parameters: criteria for the intercept and slope are dependent

27. where A is the accuracy specification of the new assay
Equivalence test
where A is the accuracy specification of the new assay
Methods:
If the 90% two-sided confidence interval of the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent
Deming-Jackknife is used to do the estimation

28. where  = Var()/Var() (assumed known or to be estimated)
Deming regression: (a.k.a. errors-in-variables regression, a structural or functional relationship model)
Minimize the sum of squares:
where  = Var()/Var() (assumed known or to be estimated)
The solutions are given by:
Weighted Deming regression:

29. Estimation of  in Deming regression
Duplicate measurements:
>2 replicates: residual errors by ANOVA
Mis-specification of  (Linnet 1998):
biased slope estimate
large Type I error

30. Jackknife estimation: to obtain the final parameter estimates and the SEs
Omit one pair of data at a time, obtain the Deming-regression estimates:
The ith pseudo-values of the intercept and slope are:
Final estimates and SEs for  and  are the mean and standard error of i and i

31. Bias estimation by Jackknife
At each nominal level , the ith pseudo-value of the Bias is:
The bias estimate and the SE at each nominal level are the mean and SE of Biasi
The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A)
The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A)

32. Example 1: methods comparison for two HIV-1 assays

33. Bland-Altman plot: potential outliers in the data

34. Identify outliers: fitting a linear regression line to the Bland Altman plot

35. Remove outliers: Bland-Altman plot shows no trend in Difference vs
Remove outliers: Bland-Altman plot shows no trend in Difference vs. Average
mean difference = 0.02
(95% CI: -0.06, 0.10)
slope = (p-value = 0.5)

36. Regression analysis: results from the 3 methods are very similar

37. Bias estimation: almost all 90% CI lie within the tolerance bounds (-0

38. Example 2: to show matrix equivalency between EDTA Plasma and Serum

39. mean difference = -0.06 (95% CI: -0.16, 0.04)
Bland-Altman plot on average titer: most titers higher than 1E5 IU/mL, heteroscedasticity?
slope = 0.03 (p-value = 0.6)
mean difference = (95% CI: -0.16, 0.04)

40. Checking for heteroscedasticity: residual errors from random effects models

41.   1: Pooled within-sample SD for EDTA = 0
  1: Pooled within-sample SD for EDTA = Pooled within-sample SD for Serum =

42. Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

43. References
Bland M., Altman D. (1986). ‘Statistical methods for assessing agreement between two methods of clinical measurement’. Lancet 347:
Hawkins D. (2002). ‘Diagnostics for conformity of paired quantitative measurements’. Stat in Med 21:
Lin L.K. (1989). ‘A concordance correlation coefficient to evaluate reproducibility’. Biometrics 45:
Linnet K. (1990). ‘Estimation of the linear relationship between the measurements of two methods with proportional bias’. Stat in Med 9:
Linnet K. (1993). ‘Evaluation of regression procedures for methods comparison studies’. Clin Chem 39:
Linnet K. (1998). ‘Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies’. Clin Chem 44:
Linnet K. (1999). ‘Necessary sample size for method comparison studies based on regression analysis’. Clin Chem 45:

44. References (continued)
Passing H., Bablok W. (1983). ‘A new biometrical procedure for testing the equality of measurements from two different analytical methods’. J Clin Chem Clin Biochem 21:
Passing H., Bablok W. (1984). ‘Comparison of several regression procedures for method comparison studies and determination of sample sizes’. J Clin Chem Clin Biochem 22:
St. Laurent R.T. (1998). ‘Evaluating Agreement with a Gold Standard in Method Comparison Studies’. Biometrics 54: