Presentation is loading. Please wait.

Presentation is loading. Please wait.

Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Similar presentations


Presentation on theme: "Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003."— Presentation transcript:

1 Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003

2 Diagnostics 2 Outline Introduction PCR based quantitative nucleic acid assays Literature references Acceptance criteria Examples References Introduction PCR based quantitative nucleic acid assays Literature references Acceptance criteria Examples References

3 Diagnostics 3 Methods Comparison Studies: to validate a new assay Purposes: To show that the new assay has good agreement with the reference assays To show that the assay performs similarly with different types of specimen Premises of methods comparison studies: A linear relationship between the two assays LOD, dynamic range have to be already established Appropriate transformation to normalize the data Analysis: To detect constant bias and proportional bias Purposes: To show that the new assay has good agreement with the reference assays To show that the assay performs similarly with different types of specimen Premises of methods comparison studies: A linear relationship between the two assays LOD, dynamic range have to be already established Appropriate transformation to normalize the data Analysis: To detect constant bias and proportional bias

4 Diagnostics 4 Constant Bias: the difference between the two methods is constant across the data range

5 Diagnostics 5 Proportional Bias: the difference between the two methods is linear across the data range

6 Diagnostics 6 PCR based nucleic acid assays To quantify the viral load by PCR method Characteristics: A wide dynamic range (e.g. 10cp/mL to 1E7 cp/mL) Skewed distribution (non-normal): typically log10 transformation for the data Heteroscedasticity: variance is higher at higher titer levels log10 transformation may not achieve homogeneity in variance (variance at lower end may increase) Other transformation: To quantify the viral load by PCR method Characteristics: A wide dynamic range (e.g. 10cp/mL to 1E7 cp/mL) Skewed distribution (non-normal): typically log10 transformation for the data Heteroscedasticity: variance is higher at higher titer levels log10 transformation may not achieve homogeneity in variance (variance at lower end may increase) Other transformation:

7 Diagnostics 7 PCR based assays: a wide dynamic range - data are log10 transformed

8 Diagnostics 8 PCR based assays: log10 transformation may remove some skewness Untransformed log10 transformed

9 Diagnostics 9 Literature references on Methods Comparison Studies Correlation coefficient Other coefficients T-test Bland-Altman plot Ordinary least squares regression Passing-Bablok regression Deming regression Correlation coefficient Other coefficients T-test Bland-Altman plot Ordinary least squares regression Passing-Bablok regression Deming regression

10 Diagnostics 10 Correlation coefficient R or R 2 Measures the strength of linear relationship between two assays Does not measure agreement: cannot detect constant or proportional bias Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995? Measures the strength of linear relationship between two assays Does not measure agreement: cannot detect constant or proportional bias Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

11 Diagnostics 11 Other coefficients Concordance coefficient (Lin, 1989) : Measures the strength of relationship between two assays that fall on the 45 o line through the origin Gold-standard correlation coefficient (St.Laurent 1998) : Measures the agreement between a new assay and a gold standard Concordance coefficient (Lin, 1989) : Measures the strength of relationship between two assays that fall on the 45 o line through the origin Gold-standard correlation coefficient (St.Laurent 1998) : Measures the agreement between a new assay and a gold standard

12 Diagnostics 12 T-test Paired t-test on the difference in the measurements by two assays Can only detect constant bias Cannot detect proportional bias Paired t-test on the difference in the measurements by two assays Can only detect constant bias Cannot detect proportional bias

13 Diagnostics 13 Bland-Altman graphical analysis (Bland and Altman, 1986) Methods: Plot the Difference of the two assays (D = X-Y) vs. the Average of the two assays (A = (X+Y)/2) Visually inspect the plot and see if there are any trends in the plot proportional bias Summarize the bias between the two assays by the mean, SD, 95% CI constant bias Modification: regress D with A, test if slope = 0 (Hawkins, 2002) A useful visual tool: transformation, heteroscedasticity, outliers, curvature Methods: Plot the Difference of the two assays (D = X-Y) vs. the Average of the two assays (A = (X+Y)/2) Visually inspect the plot and see if there are any trends in the plot proportional bias Summarize the bias between the two assays by the mean, SD, 95% CI constant bias Modification: regress D with A, test if slope = 0 (Hawkins, 2002) A useful visual tool: transformation, heteroscedasticity, outliers, curvature

14 Diagnostics 14 Bland Altman plot (continued)

15 Diagnostics 15 Ordinary least-squares regression Methods: Regress the observed data of the new assay (Y) with those of the reference assay (X) Minimize the squared deviations from the identity line in the vertical direction Modifications: weighted least squares Assumptions: The reference assay (X) is error free, or the error is relatively small compared to the range of the measurements e.g. in clinical chemistry studies, the measurement errors are minimal Methods: Regress the observed data of the new assay (Y) with those of the reference assay (X) Minimize the squared deviations from the identity line in the vertical direction Modifications: weighted least squares Assumptions: The reference assay (X) is error free, or the error is relatively small compared to the range of the measurements e.g. in clinical chemistry studies, the measurement errors are minimal

16 Diagnostics 16 Ordinary least-squares regression (continued) If measurement errors exist in both assays, the estimates are biased slope tends to be smaller intercept tends to be larger If measurement errors exist in both assays, the estimates are biased slope tends to be smaller intercept tends to be larger

17 Diagnostics 17 Passing-Bablok regression (Passing and Bablok, 1983) A nonparametric approach - robust to outliers Methods: Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate) Confidence intervals by the rank techniques Assumptions: The measurement errors in both assays follow the same type of distribution (not necessarily normal) The ratio of the variance is a constant (variance not necessarily constant across the range of data) The sampling distributions of the samples are arbitrary A nonparametric approach - robust to outliers Methods: Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate) Confidence intervals by the rank techniques Assumptions: The measurement errors in both assays follow the same type of distribution (not necessarily normal) The ratio of the variance is a constant (variance not necessarily constant across the range of data) The sampling distributions of the samples are arbitrary

18 Diagnostics 18 Deming regression (Linnet, 1990) Methods: Orthogonal least squares estimates: minimize the squared deviation of the observed data from the regression line Standard errors for the estimates obtained by Jackknife method Weighted Deming regression when heteroscedastic Assumptions: Measurement errors for both assays follow independent normal distributions with mean 0 Error variances are assumed to be proportional (variance not necessarily constant across the range of data) Methods: Orthogonal least squares estimates: minimize the squared deviation of the observed data from the regression line Standard errors for the estimates obtained by Jackknife method Weighted Deming regression when heteroscedastic Assumptions: Measurement errors for both assays follow independent normal distributions with mean 0 Error variances are assumed to be proportional (variance not necessarily constant across the range of data)

19 Diagnostics 19 Comparison of the 3 regression methods (Linnet, 1993) Electrolyte study (homogeneous variance): OLS, Passing-Bablok: biased slope, large Type I error, larger RMSE than Deming Deming: unbiased slope, correct Type I error Metabolite study (heterogeneous variance): All have unbiased slope estimates Weighted LS and weighted Deming are most efficient Type I error is large for OLS, weighted LS, Deming and Passing-Bablok Presence of outliers: Passing-Bablok is robust to outliers Deming regression requires detection of outliers Electrolyte study (homogeneous variance): OLS, Passing-Bablok: biased slope, large Type I error, larger RMSE than Deming Deming: unbiased slope, correct Type I error Metabolite study (heterogeneous variance): All have unbiased slope estimates Weighted LS and weighted Deming are most efficient Type I error is large for OLS, weighted LS, Deming and Passing-Bablok Presence of outliers: Passing-Bablok is robust to outliers Deming regression requires detection of outliers

20 Diagnostics 20 Software Statistical packages: SAS, Splus Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression) : Analyse-it (Excel add-on): does not support weighted Deming regression Method Validator (a freeware) CBStat (Linnet K.) Statistical packages: SAS, Splus Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression) : Analyse-it (Excel add-on): does not support weighted Deming regression Method Validator (a freeware) CBStat (Linnet K.)

21 Diagnostics 21 Acceptance criteria for regression type analysis Independent acceptance criteria for slope and intercept estimates: e.g. slope estimate within (0.9, 1.1), intercept estimate within (-0.2, 0.2) Drawback: asymmetrical acceptance region across the data range Independent acceptance criteria for slope and intercept estimates: e.g. slope estimate within (0.9, 1.1), intercept estimate within (-0.2, 0.2) Drawback: asymmetrical acceptance region across the data range

22 Diagnostics 22 Asymmetrical acceptance region

23 Diagnostics 23 Proposed acceptance criteria Goals: to show that the new assay is equivalent to the reference assay to demonstrate that the bias between the two assays is within some acceptable threshold across the clinical range Acceptance Criteria: Choice of tolerance level A: accuracy specification for the new assay Goals: to show that the new assay is equivalent to the reference assay to demonstrate that the bias between the two assays is within some acceptable threshold across the clinical range Acceptance Criteria: Choice of tolerance level A: accuracy specification for the new assay

24 Diagnostics 24 Mathematical models

25 Diagnostics 25 Comparison of the acceptance criteria: {Int (-0.2,0.2), Slope (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

26 Diagnostics 26 Acceptance region for the parameters: criteria for the intercept and slope are dependent

27 Diagnostics 27 Equivalence test Methods: If the 90% two-sided confidence interval of the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent Deming-Jackknife is used to do the estimation Methods: If the 90% two-sided confidence interval of the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent Deming-Jackknife is used to do the estimation where A is the accuracy specification of the new assay

28 Diagnostics 28 Deming regression: (a.k.a. errors-in-variables regression, a structural or functional relationship model) Minimize the sum of squares: The solutions are given by: where = Var( )/Var( ) (assumed known or to be estimated) Weighted Deming regression:

29 Diagnostics 29 Estimation of in Deming regression Duplicate measurements: >2 replicates: residual errors by ANOVA Mis-specification of (Linnet 1998) : biased slope estimate large Type I error Duplicate measurements: >2 replicates: residual errors by ANOVA Mis-specification of (Linnet 1998) : biased slope estimate large Type I error

30 Diagnostics 30 Jackknife estimation: to obtain the final parameter estimates and the SEs Omit one pair of data at a time, obtain the Deming-regression estimates: The i th pseudo-values of the intercept and slope are: Final estimates and SEs for and are the mean and standard error of i and i

31 Diagnostics 31 Bias estimation by Jackknife The bias estimate and the SE at each nominal level are the mean and SE of Bias i The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A) The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A) The bias estimate and the SE at each nominal level are the mean and SE of Bias i The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A) The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A) At each nominal level, the i th pseudo-value of the Bias is:

32 Diagnostics 32 Example 1: methods comparison for two HIV-1 assays

33 Diagnostics 33 Bland-Altman plot: potential outliers in the data

34 Diagnostics 34 Identify outliers: fitting a linear regression line to the Bland Altman plot

35 Diagnostics 35 Remove outliers: Bland-Altman plot shows no trend in Difference vs. Average slope = (p-value = 0.5) mean difference = 0.02 (95% CI: -0.06, 0.10)

36 Diagnostics 36 Regression analysis: results from the 3 methods are very similar

37 Diagnostics 37 Bias estimation: almost all 90% CI lie within the tolerance bounds (-0.2, +0.2)

38 Diagnostics 38 Example 2: to show matrix equivalency between EDTA Plasma and Serum

39 Diagnostics 39 Bland-Altman plot on average titer: most titers higher than 1E5 IU/mL, heteroscedasticity? slope = 0.03 (p-value = 0.6) mean difference = (95% CI: -0.16, 0.04)

40 Diagnostics 40 Checking for heteroscedasticity: residual errors from random effects models

41 Diagnostics 41 1: Pooled within-sample SD for EDTA = Pooled within-sample SD for Serum =

42 Diagnostics 42 Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

43 Diagnostics 43 References Bland M., Altman D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 347: Hawkins D. (2002). Diagnostics for conformity of paired quantitative measurements. Stat in Med 21: Lin L.K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45: Linnet K. (1990). Estimation of the linear relationship between the measurements of two methods with proportional bias. Stat in Med 9: Linnet K. (1993). Evaluation of regression procedures for methods comparison studies. Clin Chem 39: Linnet K. (1998). Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies. Clin Chem 44: Linnet K. (1999). Necessary sample size for method comparison studies based on regression analysis. Clin Chem 45: Bland M., Altman D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 347: Hawkins D. (2002). Diagnostics for conformity of paired quantitative measurements. Stat in Med 21: Lin L.K. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics 45: Linnet K. (1990). Estimation of the linear relationship between the measurements of two methods with proportional bias. Stat in Med 9: Linnet K. (1993). Evaluation of regression procedures for methods comparison studies. Clin Chem 39: Linnet K. (1998). Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies. Clin Chem 44: Linnet K. (1999). Necessary sample size for method comparison studies based on regression analysis. Clin Chem 45:

44 Diagnostics 44 References (continued) Passing H., Bablok W. (1983). A new biometrical procedure for testing the equality of measurements from two different analytical methods. J Clin Chem Clin Biochem 21: Passing H., Bablok W. (1984). Comparison of several regression procedures for method comparison studies and determination of sample sizes. J Clin Chem Clin Biochem 22: St. Laurent R.T. (1998). Evaluating Agreement with a Gold Standard in Method Comparison Studies. Biometrics 54: Passing H., Bablok W. (1983). A new biometrical procedure for testing the equality of measurements from two different analytical methods. J Clin Chem Clin Biochem 21: Passing H., Bablok W. (1984). Comparison of several regression procedures for method comparison studies and determination of sample sizes. J Clin Chem Clin Biochem 22: St. Laurent R.T. (1998). Evaluating Agreement with a Gold Standard in Method Comparison Studies. Biometrics 54:


Download ppt "Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003."

Similar presentations


Ads by Google