Presentation is loading. Please wait.

Presentation is loading. Please wait.

ROC Analysis Emily Kistner-Griffin, PhD Amy Wahlquist, MS Cancer Prevention and Control Statistics Tutorial August 13, 2009.

Similar presentations

Presentation on theme: "ROC Analysis Emily Kistner-Griffin, PhD Amy Wahlquist, MS Cancer Prevention and Control Statistics Tutorial August 13, 2009."— Presentation transcript:

1 ROC Analysis Emily Kistner-Griffin, PhD Amy Wahlquist, MS Cancer Prevention and Control Statistics Tutorial August 13, 2009

2 Outline I.Motivating Example: Chest CT II.Classification III.Sensitivity and Specificity IV.ROC curve and AUC estimation a.Nonparametric Curve b.Parametric Curve V.ROC and Logistic Regression VI.Comparing ROC curves

3 I. Motivating Example: Chest CT Evaluating the probability of malignancy in pulmonary nodules seen on chest CT in 213 MUSC patients from two cohorts Sample of 194 subjects seen in pulmonary clinic and 19 subjects with CT previous to an unrelated surgical intervention Develop a prediction model from clinical data and radiological characteristics of lung nodules

4 Chest CT A model of P (malignancy) of pulmonary nodules has been described in the literature (Swensen SJ et al., 1997) Model included three demographic characteristics: patient age, smoking status (ever vs. never), any history of cancer Model included three radiological characteristics: diameter, upper lobe location, and spiculation

5 Chest CT Swensen et al. reported an area under the reciever operating curve of ± in a validation sample, using a logistic regression approach. Interested in how well Swensen’s model performs in the MUSC cohort. Interested in evaluating whether we can improve the prediction model by including other patient characteristics

6 II. Classification Consider medical tests that are measured on a continuous or ordinal scale Goal: to describe the performance of the medical test in classifying subjects into individuals with and without disease Examples: PSA and CA-125 as biomarkers of prostate and ovarian cancer; BI-RADS for breast imaging (radiologist determined probability of malignancy)

7 Classification from CT Consider the diameter of the nodule as measured on the CT scan (range: 3.3mm-15mm) Larger nodules are more likely to be malignant (OR: 1.34, 95% CI: ) How well can we predict malignancy from nodule diameter?

8 Contingency Table d<66≤d<88≤d<1010≤d<1212≤d≤15 Benign Malignant

9 Classification Tables Choose a cut-point on continuous or ordinal scale in order to assign disease status Truth D=1 Truth D=0 Classified D=1 TPFP Classified D=0 FNTN

10 III. Sensitivity & Specificity For selected cut-point determine sensitivity and specificity of medical test (or prediction model) Sensitivity = Pr ( TP | + ) = TP / (TP+FN) = TPF Specificity = Pr ( TN | — ) = TN / (TN+FP) = TNF In order to summarize test characteristics – must compute sensitivity and specificity at multiple cut- points

11 Sensitivity & Specificity Example Cut-pointSensitivitySpecificity

12 From Metz CE (1978) Basic Principles of ROC Analysis. Seminars in Nuclear Medicine; 8 (4): 283 – 297.



15 Decision Threshold Lowering the threshold increases TPF (sensitivity) and the FPF (1-specificity) Raising the threshold decreases the TPF and the FPF Points representing all possible TPF and FPF lie on a curve – passing through the lower (0,0) corner when all tests are called negative and the upper (1,1) corner when all the tests are called positive If the test is informative then all other points on the curve must be above the diagonal (TP more likely than FP) The curve describing the compromises between TPF and FPF is called the ROC curve


17 Detailed report of Sensitivity and Specificity Correctly Cutpoint Sensitivity Specificity Classified LR+ LR ( >= 3.3 ) % 0.00% 33.80% ( >= 4 ) % 1.42% 34.74% ( >= 5 ) 97.22% 13.48% 41.78% ( >= 6 ) 97.22% 29.08% 52.11% ( >= 7 ) 93.06% 39.72% 57.75% ( >= 8 ) 83.33% 50.35% 61.50% ( >= 9.1 ) 70.83% 61.70% 64.79% ( >= 10 ) 65.28% 70.92% 69.01% ( >= 11 ) 56.94% 78.72% 71.36% ( >= 12 ) 45.83% 82.98% 70.42% ( >= 13 ) 25.00% 90.78% 68.54% ( >= 14 ) 13.89% 95.04% 67.61% ( >= 15 ) 0.00% 98.58% 65.26% ( > 15 ) 0.00% % 66.20% roctab malignant diameter, detail graph ROC -Asymptotic Normal-- Obs Area Std. Err. [95% Conf. Interval]

18 Likelihood Ratios LR+ = sensitivity / (1-specificity) = TPF FPF LR- = (1-sensitivity) / specificity = 1-TPF 1-FPF LR+ is the slope between the origin and the point on the ROC curve and LR- is the slope between the point on the curve and the (1,1) point (Choi 1998)

19 IV. ROC curve and AUC estimation ROC: Receiver Operating Characteristic Developed in signal detection theory to illustrate how the receiver deciphers between signal and noise (1960s) Illustration of two test characteristics: sensitivity and specificity at selected cut-points (decision thresholds) Popularized in medical testing in the field of Radiology (1980s)

20 ROC curve and Thresholds ROC curve describes disease detection independent of disease prevalence (sensitivity and specificity are also) Prevalence may help determine the operating threshold: –Low prevalence suggests reducing FPF (higher specificity, higher threshold, lower part of the curve) –High prevalence suggests increasing TPF (higher sensitivity, lower threshold, higher part of the curve) In practice, must consider costs and consequences of FP and FN before selecting the desirable cut-off: –Consequence of FN: death? –Consequence of FP: stressful, costly work-up or treatment

21 Area Under the ROC Curve Summarizes the performance of the test Probability that the result of the test for a randomly selected abnormal subject will be greater than the result of the test for a randomly selected normal subject Average TPF: averaged across whole range of FPF in (0,1) Perfect test gives AUC = 1.0 and an uninformative test gives AUC=0.50 Parametric and non-parametric approaches to constructing the ROC curve and calculating the area under the curve (AUC)

22 a. Nonparametric ROC Curve Constructed by plotting sensitivity and (1 – specificity) at each possible cut-point Area under the curve (AUC) constructed using the trapezoidal rule Variance estimators have been derived Delong et al. (1988), Hanley and McNeil (1982); Bamber (1975)

23 Variance of AUC Specifically for Delong et al. (1988) variance estimate:

24 Confidence Intervals for AUC Must consider distribution of AUC estimate: asymptotically normal or binomial assumption Must select standard error estimate (Delong et al. approach is the default): ROC -Asymptotic Normal-- Obs Area Std. Err. [95% Conf. Interval] roctab malignant diameter, binomial ROC -- Binomial Exact -- Obs Area Std. Err. [95% Conf. Interval]

25 b. Parametric ROC Curve Assumes a binormal model A monotone transformation of the test results exists to give results that are normally distributed in the diseased and non- diseased populations Method involves fitting a straight line to the empirical ROC points by plotting using normal probability scales on each axis (plot inverse of the standard normal cumulative distribution function for sensitivity and specificity) Intercept of the line is the standardized difference in the continuous variable between the two populations; slope is a ratio of the standard deviations

26 Parametric AUC Estimation AUC is a function of the slope and intercept of the estimated line – using the standard normal cumulative distribution function

27 Nonparametric vs. Parametric Parametric approaches assume a binormal distribution to makes inferences (obtain MLE): only when the assumption is true are the estimators unbiased With continuous data a nonparametric approach is recommended With discrete ratings a parametric approach is recommended as nonparametric approaches tend to underestimate the true AUC Note standard error of the AUC is smaller using a continuous scale

28 . rocfit malignant diameter, cont(10) Fitting binormal model: Binormal model of malignant on diameter Number of obs = 213 Goodness-of-fit chi2(7) = 8.52 Prob > chi2 = Log likelihood = | Coef. Std. Err. z P>|z| [95% Conf. Interval] intercept | slope (*) | /cut1 | /cut2 | /cut3 | /cut4 | /cut5 | /cut6 | /cut7 | /cut8 | /cut9 | | Indices from binormal fit Index | Estimate Std. Err. [95% Conf. Interval] ROC area | delta(m) | d(e) | d(a) | (*) z test for slope==1

29 . rocplot, confband

30 V. ROC and Logistic Regression Prediction Model from Chest CT Use logistic regression to create probabilities of malignancy (represent diagnostic results from multiple predictors) Compare two logistic models of malignancy – one from previous literature and model with selected variables from the MUSC data Variables suggested in Swensen SJ et al. + surgical cohort (variable describing collection of samples) Variables selected using backwards regression in MUSC data

31 . logistic malignant surgical_cohort patient_age any_non_lung_cancer_history lung_cancer_history smoker_ever diameter upper_lobe spiculated Logistic regression Number of obs = 207 LR chi2(8) = Prob > chi2 = Log likelihood = Pseudo R2 = malignant | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] surgical_c~t | patient_age | any_non_lu~y | lung_cance~y | smoker_ever | diameter | upper_lobe | spiculated | predict swensen

32 . lsens, gensens(sensitivity) genspec(specificity) genpr(cutoffs)

33 . lroc

34 Use saved predicted probabilities from logistic model:. roctab malignant swensen ROC -Asymptotic Normal-- Obs Area Std. Err. [95% Conf. Interval] roctab malignant swensen, graph Postestimation: 95% CI

35 Again use saved predicted probabilities from logistic model:. roccomp malignant diameter swensen ROC -Asymptotic Normal-- Obs Area Std. Err. [95% Conf. Interval] diameter swensen Ho: area(diameter) = area(swensen) chi2(1) = 9.52 Prob>chi2 = VI. Comparing ROC curves

36 Using quantities defined by Delong et al. for variance estimation to define chi-squared test statistic: Testing AUC Equality

37 Models with Multiple Predictors. logistic malignant diameter any_non_lung_cancer_history surgical_cohort lung_cancer_history pet_positive pack Logistic regression Number of obs = 206 LR chi2(6) = Prob > chi2 = Log likelihood = Pseudo R2 = malignant | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval] diameter | any_non_lu~y | surgical_c~t | lung_cance~y | pet_positive | pack | predict musc

38 . roccomp malignant diameter swensen musc, graph summary

39 . roccomp malignant diameter swensen musc ROC -Asymptotic Normal-- Obs Area Std. Err. [95% Conf. Interval] diameter swensen musc Ho: area(diameter) = area(swensen) = area(musc) chi2(2) = Prob>chi2 = rocgold malignant swensen diameter musc ROC Bonferroni Area Std. Err. chi2 df Pr>chi2 Pr>chi swensen (standard) diameter musc

40 Questions? Next: ROC in SPSS

41 b. Lorenz Curves ROC curve represents a monotone increasing function of the FPF (1-specificity) If the risk of disease does not vary monotonically with the diagnostic test then the ROC may not be convex Lee (1999) suggested a Lorenz curve (used commonly in economics) for such data The methodology involves reordering the test results to ensure that the ratio of disease subjects / no disease subjects in each category is increasing Must consider whether reordering makes practical sense (usually sensible on an ordinal scale but not necessarily on a continuous scale)


43 Defining Lorenz Curves Plot cumulative percent of individuals with disease against the cumulative percent of individuals without the disease at each cut-point Examples when a Lorenz might be appropriate: –Test has similar means but different variances across populations with and without disease –Bimodal distribution of test in either population –Skewed distribution in population with disease and symmetric distribution in population without the disease A flatter Lorenz curve suggests a worse diagnostic test Two summary indices describe the curvature –Gini index: twice the area between the Lorenz curve and the diagonal line –Pietra index: twice the area of the largest triangle inscribed between the diagonal line and the curve

44 Lorenz Curves and ROC. roctab malignant diameter, lorenz graph. roctab malignant diameter, lorenz Lorenz curve Pietra index = Gini index = If the at-risk probabilities increase (or decrease) with increasing values of the test results then Gini = 2(AUC)-1 Larger Pietra and Gini indices describe better diagnostic tests Gini index is related to average difference in post-test probabilities for two randomly selected subjects and Pietra index is related to average absolute change between pre and post test probabilities of disease

Download ppt "ROC Analysis Emily Kistner-Griffin, PhD Amy Wahlquist, MS Cancer Prevention and Control Statistics Tutorial August 13, 2009."

Similar presentations

Ads by Google