Presentation is loading. Please wait.

Presentation is loading. Please wait.

Methods Comparison Studies for Quantitative Nucleic Acid Assays

Similar presentations

Presentation on theme: "Methods Comparison Studies for Quantitative Nucleic Acid Assays"— Presentation transcript:

1 Methods Comparison Studies for Quantitative Nucleic Acid Assays
Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003

2 Outline Introduction PCR based quantitative nucleic acid assays
Literature references Acceptance criteria Examples References

3 Methods Comparison Studies: to validate a new assay
Purposes: To show that the new assay has good agreement with the reference assays To show that the assay performs similarly with different types of specimen Premises of methods comparison studies: A linear relationship between the two assays LOD, dynamic range have to be already established Appropriate transformation to normalize the data Analysis: To detect constant bias and proportional bias

4 Constant Bias: the difference between the two methods is constant across the data range

5 Proportional Bias: the difference between the two methods is linear across the data range

6 PCR based nucleic acid assays
To quantify the viral load by PCR method Characteristics: A wide dynamic range (e.g. 10cp/mL to 1E7 cp/mL) Skewed distribution (non-normal): typically log10 transformation for the data Heteroscedasticity: variance is higher at higher titer levels log10 transformation may not achieve homogeneity in variance (variance at lower end may increase) Other transformation:

7 PCR based assays: a wide dynamic range - data are log10 transformed

8 PCR based assays: log10 transformation may remove some skewness
Untransformed log10 transformed

9 Literature references on Methods Comparison Studies
Correlation coefficient Other coefficients T-test Bland-Altman plot Ordinary least squares regression Passing-Bablok regression Deming regression

10 Correlation coefficient R or R2
Measures the strength of linear relationship between two assays Does not measure agreement: cannot detect constant or proportional bias Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

11 Other coefficients Concordance coefficient (Lin, 1989):
Measures the strength of relationship between two assays that fall on the 45o line through the origin Gold-standard correlation coefficient (St.Laurent 1998): Measures the agreement between a new assay and a gold standard

12 T-test Paired t-test on the difference in the measurements by two assays Can only detect constant bias Cannot detect proportional bias

13 Bland-Altman graphical analysis (Bland and Altman, 1986)
Methods: Plot the Difference of the two assays (D = X-Y) vs. the Average of the two assays (A = (X+Y)/2) Visually inspect the plot and see if there are any trends in the plot  proportional bias Summarize the bias between the two assays by the mean, SD, 95% CI  constant bias Modification: regress D with A, test if slope = 0 (Hawkins, 2002) A useful visual tool: transformation, heteroscedasticity, outliers, curvature

14 Bland Altman plot (continued)

15 Ordinary least-squares regression
Methods: Regress the observed data of the new assay (Y) with those of the reference assay (X) Minimize the squared deviations from the identity line in the vertical direction Modifications: weighted least squares Assumptions: The reference assay (X) is error free, or the error is relatively small compared to the range of the measurements e.g. in clinical chemistry studies, the measurement errors are minimal

16 Ordinary least-squares regression (continued)
If measurement errors exist in both assays, the estimates are biased slope tends to be smaller intercept tends to be larger

17 Passing-Bablok regression (Passing and Bablok, 1983)
A nonparametric approach - robust to outliers Methods: Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate) Confidence intervals by the rank techniques Assumptions: The measurement errors in both assays follow the same type of distribution (not necessarily normal) The ratio of the variance is a constant (variance not necessarily constant across the range of data) The sampling distributions of the samples are arbitrary

18 Deming regression (Linnet, 1990)
Methods: Orthogonal least squares estimates: minimize the squared deviation of the observed data from the regression line Standard errors for the estimates obtained by Jackknife method Weighted Deming regression when heteroscedastic Assumptions: Measurement errors for both assays follow independent normal distributions with mean 0 Error variances are assumed to be proportional (variance not necessarily constant across the range of data)

19 Comparison of the 3 regression methods (Linnet, 1993)
Electrolyte study (homogeneous variance): OLS, Passing-Bablok: biased slope, large Type I error, larger RMSE than Deming Deming: unbiased slope, correct Type I error Metabolite study (heterogeneous variance): All have unbiased slope estimates Weighted LS and weighted Deming are most efficient Type I error is large for OLS, weighted LS, Deming and Passing-Bablok Presence of outliers: Passing-Bablok is robust to outliers Deming regression requires detection of outliers

20 Software Statistical packages: SAS, Splus
Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression): Analyse-it (Excel add-on): does not support weighted Deming regression Method Validator (a freeware) CBStat (Linnet K.)

21 Acceptance criteria for regression type analysis
Independent acceptance criteria for slope and intercept estimates: e.g. slope estimate within (0.9, 1.1), intercept estimate within (-0.2, 0.2) Drawback: asymmetrical acceptance region across the data range

22 Asymmetrical acceptance region

23 Proposed acceptance criteria
Goals: to show that the new assay is ‘equivalent’ to the reference assay to demonstrate that the bias between the two assays is within some acceptable threshold across the clinical range Acceptance Criteria: Choice of tolerance level A: accuracy specification for the new assay

24 Mathematical models

25 Comparison of the acceptance criteria: {Int  (-0. 2,0. 2), Slope  (0
Comparison of the acceptance criteria: {Int  (-0.2,0.2), Slope  (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

26 Acceptance region for the parameters: criteria for the intercept and slope are dependent

27 where A is the accuracy specification of the new assay
Equivalence test where A is the accuracy specification of the new assay Methods: If the 90% two-sided confidence interval of the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent Deming-Jackknife is used to do the estimation

28 where  = Var()/Var() (assumed known or to be estimated)
Deming regression: (a.k.a. errors-in-variables regression, a structural or functional relationship model) Minimize the sum of squares: where  = Var()/Var() (assumed known or to be estimated) The solutions are given by: Weighted Deming regression:

29 Estimation of  in Deming regression
Duplicate measurements: >2 replicates: residual errors by ANOVA Mis-specification of  (Linnet 1998): biased slope estimate large Type I error

30 Jackknife estimation: to obtain the final parameter estimates and the SEs
Omit one pair of data at a time, obtain the Deming-regression estimates: The ith pseudo-values of the intercept and slope are: Final estimates and SEs for  and  are the mean and standard error of i and i

31 Bias estimation by Jackknife
At each nominal level , the ith pseudo-value of the Bias is: The bias estimate and the SE at each nominal level are the mean and SE of Biasi The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A) The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A)

32 Example 1: methods comparison for two HIV-1 assays

33 Bland-Altman plot: potential outliers in the data

34 Identify outliers: fitting a linear regression line to the Bland Altman plot

35 Remove outliers: Bland-Altman plot shows no trend in Difference vs
Remove outliers: Bland-Altman plot shows no trend in Difference vs. Average mean difference = 0.02 (95% CI: -0.06, 0.10) slope = (p-value = 0.5)

36 Regression analysis: results from the 3 methods are very similar

37 Bias estimation: almost all 90% CI lie within the tolerance bounds (-0

38 Example 2: to show matrix equivalency between EDTA Plasma and Serum

39 mean difference = -0.06 (95% CI: -0.16, 0.04)
Bland-Altman plot on average titer: most titers higher than 1E5 IU/mL, heteroscedasticity? slope = 0.03 (p-value = 0.6) mean difference = (95% CI: -0.16, 0.04)

40 Checking for heteroscedasticity: residual errors from random effects models

41   1: Pooled within-sample SD for EDTA = 0
  1: Pooled within-sample SD for EDTA = Pooled within-sample SD for Serum =

42 Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

43 References Bland M., Altman D. (1986). ‘Statistical methods for assessing agreement between two methods of clinical measurement’. Lancet 347: Hawkins D. (2002). ‘Diagnostics for conformity of paired quantitative measurements’. Stat in Med 21: Lin L.K. (1989). ‘A concordance correlation coefficient to evaluate reproducibility’. Biometrics 45: Linnet K. (1990). ‘Estimation of the linear relationship between the measurements of two methods with proportional bias’. Stat in Med 9: Linnet K. (1993). ‘Evaluation of regression procedures for methods comparison studies’. Clin Chem 39: Linnet K. (1998). ‘Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies’. Clin Chem 44: Linnet K. (1999). ‘Necessary sample size for method comparison studies based on regression analysis’. Clin Chem 45:

44 References (continued)
Passing H., Bablok W. (1983). ‘A new biometrical procedure for testing the equality of measurements from two different analytical methods’. J Clin Chem Clin Biochem 21: Passing H., Bablok W. (1984). ‘Comparison of several regression procedures for method comparison studies and determination of sample sizes’. J Clin Chem Clin Biochem 22: St. Laurent R.T. (1998). ‘Evaluating Agreement with a Gold Standard in Method Comparison Studies’. Biometrics 54:

Download ppt "Methods Comparison Studies for Quantitative Nucleic Acid Assays"

Similar presentations

Ads by Google