1. Receiver Operating Characteristic Curve (ROC) Analysis for Prediction Studies Ruth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg Receiver Operating Characteristic Curve (ROC) Analysis for Prediction Studies Ruth O’Hara, Helena Kraemer, Jerome Yesavage, Jean Thompson, Art Noda, Joy Taylor, Jared Tinklenberg Stanford University, Department of Psychiatry and Behavioral Sciences Stanford University School of Medicine Sierra Pacific MIRECC Veterans Affairs Palo Alto Health Care System

2. Clinical practice is often “hit or miss” therapy Clinical practice is often “hit or miss” therapy Try one thing, if that does not work, try another Try one thing, if that does not work, try another This is frustrating for the patient and expensive This is frustrating for the patient and expensive The Goal: find the best treatment for the patient with specific characteristics The Goal: find the best treatment for the patient with specific characteristics New news in psychiatry; old hat in internal medicine New news in psychiatry; old hat in internal medicine The Clinical Need for Signal Detection Procedures

3. Receiver Operating Characteristic Curve (ROC) Analysis Signal Detection Technique Signal Detection Technique Traditionally used to evaluate diagnostic tests Traditionally used to evaluate diagnostic tests Now employed to identify subgroups of a population at differential risk for a specific outcome (clinical decline, treatment response) Now employed to identify subgroups of a population at differential risk for a specific outcome (clinical decline, treatment response) Identifies moderators Identifies moderators

4. Receiver Operating Characteristic Curve (ROC) Analysis Historical Development

5. ROC Analysis: Historical Development (1) Derived from early radar in WW2 Battle of Britain to address: Accurately identifying the signals on the radar scan to predict the outcome of interest – Enemy planes – when there were many extraneous signals (e.g. Geese)? Derived from early radar in WW2 Battle of Britain to address: Accurately identifying the signals on the radar scan to predict the outcome of interest – Enemy planes – when there were many extraneous signals (e.g. Geese)?

6. ROC Analysis: Historical Development (2) True Positives = Radar Operator interpreted signal as Enemy Planes and there were Enemy planes (Good Result: No wasted Resources) True Positives = Radar Operator interpreted signal as Enemy Planes and there were Enemy planes (Good Result: No wasted Resources) True Negatives = Radar Operator said no planes and there were none (Good Result: No wasted resources) True Negatives = Radar Operator said no planes and there were none (Good Result: No wasted resources) False Positives = Radar Operator said planes, but there were none (Geese: wasted resources) False Positives = Radar Operator said planes, but there were none (Geese: wasted resources) False Negatives = Radar Operator said no plane, but there were planes (Bombs dropped: very bad outcome) False Negatives = Radar Operator said no plane, but there were planes (Bombs dropped: very bad outcome)

7. ROC Analysis: Historical Development Sensitivity = Probability of correctly interpreting the radar signal as Enemy planes among those times when Enemy planes were actually coming Sensitivity = Probability of correctly interpreting the radar signal as Enemy planes among those times when Enemy planes were actually coming SE = True Positives / True Positives + False NegativesSE = True Positives / True Positives + False Negatives Specificity = Probability of correctly interpreting the radar signal as no Enemy planes among those times when no Enemy planes were actually coming Specificity = Probability of correctly interpreting the radar signal as no Enemy planes among those times when no Enemy planes were actually coming SP = True Negatives / True Negatives + False PositivesSP = True Negatives / True Negatives + False Positives

8. ROC: Prediction of Enemy Planes by RAF Radar Operators

9. Receiver Operating Characteristic Curve (ROC) Analysis Applications: Evaluating Medical Tests

10. ROC Analysis: Evaluating Medical Tests The evaluation of the ability of a diagnostic test to identify a disease involves considering: The evaluation of the ability of a diagnostic test to identify a disease involves considering: P=Prevalence = occurrence in the population of the outcome of interest (e.g. disease) P=Prevalence = occurrence in the population of the outcome of interest (e.g. disease) True Positives True Positives True Negatives True Negatives False Positives False Positives False Negatives False Negatives P=Prevalence=True Positives + False Negatives P=Prevalence=True Positives + False Negatives

11. ROC Analysis: Medical Test Evaluation True Positives = Test states you have the disease when you do have the disease True Positives = Test states you have the disease when you do have the disease True Negatives = Test states you do not have the disease when you do not have the disease True Negatives = Test states you do not have the disease when you do not have the disease False Positives = Test states you have the disease when you do not have the disease False Positives = Test states you have the disease when you do not have the disease False Negatives = Test states you do not have the disease when you do False Negatives = Test states you do not have the disease when you do

12. ROC Analysis: Evaluating Medical Tests Sensitivity =The probability of having a positive test result among those with a positive diagnosis for the disease Sensitivity =The probability of having a positive test result among those with a positive diagnosis for the disease SE = True Positives / True Positives + False NegativesSE = True Positives / True Positives + False Negatives Specificity = The probability of having a negative test result among those with a negative diagnosis for the disease Specificity = The probability of having a negative test result among those with a negative diagnosis for the disease SP = True Negatives / True Negatives + False PositivesSP = True Negatives / True Negatives + False Positives

13. The Basic Tool: 2X2 Test+Test- O+TP(a)FN(b) P(a + b) O-FP(c)TN(d)P'=1-P Q(a + c) Q'=1-Q Sensitivity (SE)=a/PSpecificity (SP)=d/P’

14. ROC: GDS (Test) for Diagnosis of Clinically Confirmed Depression

15. Which Test Do You Use: Medical Tests Evaluation GDS: SE =.80; SP =.85 GDS: SE =.80; SP =.85 Beck Depression Inventory: SE =.85; SP =.75 Beck Depression Inventory: SE =.85; SP =.75 Major Depression Inventory = SE =.66; SP =.63 Major Depression Inventory = SE =.66; SP =.63

16. ROC Analysis ROC first calculates Sensitivity and Specificity ROC first calculates Sensitivity and Specificity Quality Indices measures the quality of the sensitivity and specificity Quality Indices measures the quality of the sensitivity and specificity ROC computes the quality indices for each predictor to find the ones with optimal sensitivity and specificity ROC computes the quality indices for each predictor to find the ones with optimal sensitivity and specificity

17. To Detect the Optimal Sensitivity and Specificity Depends on the relative CLINICAL importance of false negatives versus false positives. W=1 means only false negatives matter. W=0 means only false positives matter. W=1/2 means both matter equally. Analytically: Use weighted kappa.

18. ROC Analysis P = TP + FNP’= 1- (TP + FN) P = TP + FNP’= 1- (TP + FN) Q = TP + FPQ’= 1- (TP + FP) Q = TP + FPQ’= 1- (TP + FP) EFF = TP + TN EFF = TP + TN κ(0.5, 0) = [ (TP + TN) - (TP + FN)(TP+FP) - (1-(TP + FN)(1-(TP + FP))] κ(0.5, 0) = [ (TP + TN) - (TP + FN)(TP+FP) - (1-(TP + FN)(1-(TP + FP))] [1 – (TP + FN)(TP+FP) - (1-(TP + FN))(1-(TP + FP))]

19. ROC Plane and “Curve” (P,P) (Q,Q) Random ROC Ideal Point ROC “curve”

20. Receiver Operating Characteristic Curve (ROC) Analysis Applications Identifying Predictors of Clinical Outcome

21. ROC Analysis: Prediction Studies (Dr. Kraemer) ROC can identify predictors/characteristics ROC can identify predictors/characteristics of patients that are at differential risk for a specific outcome of interest. e.g. What are the Characteristics of AD Patients at risk for rapid decline and are high priority for treatment? of patients that are at differential risk for a specific outcome of interest. e.g. What are the Characteristics of AD Patients at risk for rapid decline and are high priority for treatment? What are the clinical predictors of Alzheimer Disease patients who are “good responders” (or “poor responders”) to cholinesterase inhibitor treatments? What are the clinical predictors of Alzheimer Disease patients who are “good responders” (or “poor responders”) to cholinesterase inhibitor treatments? Useful in “real world” clinical medicine where multiple variables affect the clinical outcome and patients seldom have one pure diagnosis Useful in “real world” clinical medicine where multiple variables affect the clinical outcome and patients seldom have one pure diagnosis

22. ROC: Identifying Predictors of an Outcome 1. ROC relates a predictor (test) to the clinical outcome of interest (Diagnosis/Gold Standard) 1. ROC relates a predictor (test) to the clinical outcome of interest (Diagnosis/Gold Standard) 2. ROC searches all predictors and their associated cut-points 2. ROC searches all predictors and their associated cut-points 3. ROC determines which predictor and associated cut-point yields the optimal sensitivity and specificity for identifying the outcome of interest yielding two groups at differential risk for the outcome 3. ROC determines which predictor and associated cut-point yields the optimal sensitivity and specificity for identifying the outcome of interest yielding two groups at differential risk for the outcome

23. ROC: Identifying Predictors of an Outcome 4. ROC is an iterative process that is then rerun automatically for each group yielded in Step 3. in order to examine which predictor and associated cut-point may further divide the groups 4. ROC is an iterative process that is then rerun automatically for each group yielded in Step 3. in order to examine which predictor and associated cut-point may further divide the groups 5. ROC will keep searching within each group yielded until one of three stopping rules apply (see Stopping rule slide) 5. ROC will keep searching within each group yielded until one of three stopping rules apply (see Stopping rule slide) 6. ROC thus identifies subgroups of individuals that are at increased risk for the outcome of interest 6. ROC thus identifies subgroups of individuals that are at increased risk for the outcome of interest

24. ROC Analysis: Advantages and Disadvantages ROC Analysis: Advantages and Disadvantages No assumptions of normal distribution No assumptions of normal distribution Multiple predictors can be evaluated simultaneously Multiple predictors can be evaluated simultaneously Indicates interactions among predictors Indicates interactions among predictors Indicates cut-points on these predictors Indicates cut-points on these predictors Yields clinically relevant information Yields clinically relevant information Non-hypothesis testing Non-hypothesis testing Requires large samples Requires large samples Capitalizes on chance: needs stringent stopping rule Capitalizes on chance: needs stringent stopping rule

25. ROC Analysis: Procedure Start with large sample size Start with large sample size Define the outcome of interest (always binary) Define the outcome of interest (always binary) Choose Success/Failure criteria Choose Success/Failure criteria Select predictor variables of interest (as many as you like) Select predictor variables of interest (as many as you like) Run ROC Program that systematically finds best predictors for Success/Failure Run ROC Program that systematically finds best predictors for Success/Failure

26. The Basic Tool: 2X2 RF+RF- O+TP(a)FN(b) P(a + b) O-FP(c)TN(d)P'=1-P Q(a + c) Q'=1-Q Sensitivity (SE)=a/PSpecificity (SP)=d/P’

27. ROC: Identifying Predictors & Their Cut-points Dichotomous Variables such as Gender: Dichotomous Variables such as Gender: ROC calculates the Se and Sp for Female vs. MaleROC calculates the Se and Sp for Female vs. Male For Continuous Variables such as Age: For Continuous Variables such as Age: ROC would calculate Se and Sp for the cut- point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.ROC would calculate Se and Sp for the cut- point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

28. ROC: Gender as Predictor of Clinically Confirmed Depression

29. ROC: Identifying Predictors & Their Cut-points Dichotomous Variables: ROC calculates the Se and Sp for Female vs. Male, Aphasia vs. No Aphasia, etc. Dichotomous Variables: ROC calculates the Se and Sp for Female vs. Male, Aphasia vs. No Aphasia, etc. For Continuous Variables such as Age: For Continuous Variables such as Age: ROC would calculate Se and Sp for the cut- point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.ROC would calculate Se and Sp for the cut- point of 60 vs. 61+62+63 ….85; then could calculate for cut-point of 60+61 vs. 62+63+64 ….85, and so forth.

30. ROC: Age as Predictor of Clinically Confirmed Depression

32. Receiver Operating Characteristic Curve (ROC) Analysis Conducting the ROC: An Example

33. ROC Analysis: Procedure Start with large sample size Start with large sample size Define the outcome of interest Define the outcome of interest Choose Success/Failure criteria Choose Success/Failure criteria Identify predictor variables of interest Identify predictor variables of interest Run ROC Program that systematically finds best predictors for Success/Failure Run ROC Program that systematically finds best predictors for Success/Failure

34. ROC Analysis: Example Population under investigation: 1, 472 AD patients from 10 Centers with a 12 month follow-up Population under investigation: 1, 472 AD patients from 10 Centers with a 12 month follow-up Clinically significant outcome: More rapid decline as defined by a loss of 3 or more MMSE points per year, post-visit Clinically significant outcome: More rapid decline as defined by a loss of 3 or more MMSE points per year, post-visit O'Hara R et al. (2002). Which Alzheimer patients are at risk for rapid cognitive decline? J Geriatr Psychiatry Neurol;15(4):233-8.

35. Predictor Variables Age-at -patient-visit Age-at -patient-visit Reported age of symptom onset Reported age of symptom onset Gender Gender Years of education Years of education Ethnicity Ethnicity MMSE score MMSE score Living Arrangement Living Arrangement Presence of Aphasia Presence of Aphasia Presence of Hallucinations Presence of Hallucinations Presence of Extrapyramidal Signs Presence of Extrapyramidal Signs

37. Stopping Rules No more possibilities (rare!) No more possibilities (rare!) Inadequate sample size Inadequate sample size Optimal test (if ‘a priori’) would not have been statistically significant (p<.001) Optimal test (if ‘a priori’) would not have been statistically significant (p<.001)

38. Figure 10.3 P=.53 N=512 (100%) P=.53 ROC Decision Tree for IHDP Control group with outcome of low IQ at age 3. (w= 0.5) Non-minority Minority P=.70 N = 321 (63%) P=.70 P=.25 N = 191 (37%) P=.25 P=.81 N=211 (41%) P=.81 P=.48 N=110 (21%) P=.48 P=.45 N=87 (17%) P=.45 Bayley Mental Dev. Index < 115 Mother never attended college P=.09 N=104 (20%) P=.09 Mother attended college Bayley Mental Dev. Index ≥ 115 P=.91 N=131 (26%) P=.91 P=.65 N=80 (16%) P=.65 P=.30 N=57 (11%) P=.30 P=.73 N=30 (6%) P=.73 P=.19 N=43 (8%) P=.19 P=.02 N=61 (12%) P=.02 Bayley Mental Dev. Index<106 Bayley Mental Dev. Index≥106 Bayley Mental Dev. Index<106 Bayley Mental Dev. Index≥106 Graduated from college Attended, did not graduate

39. ROC Plane and “Swarm” of Points ROC “curve”

40. To Detect the Optimal Sensitivity and Specificity Depends on the relative CLINICAL importance of false negatives versus false positives. W=1 means only false negatives matter. W=0 means only false positives matter. W=1/2 means both matter equally. Analytically: Use weighted kappa. Geometrically: Draw a line through the Ideal Point with slope determined by P and w. Push this line down until it just touches the ROC “curve”. That point is optimal.

41. ROC Analysis: Conclusion Yields Clinically Relevant Information Yields Clinically Relevant Information Identifies complex interactions Identifies complex interactions Identifies individuals with different characteristics but at the same risk for the clinically relevant outcome Identifies individuals with different characteristics but at the same risk for the clinically relevant outcome Identifies individuals at the least risk Identifies individuals at the least risk Can take differential clinical costs of false positives and false negatives into account Can take differential clinical costs of false positives and false negatives into account

42. Conclusion It is not sufficient to identify risk factors or even to identify moderators and mediators etc. or a structural model. It is not sufficient to identify risk factors or even to identify moderators and mediators etc. or a structural model. It is necessary to present and interpret the results so that clinicians, policy makers, consumers, other researchers can apply them. It is necessary to present and interpret the results so that clinicians, policy makers, consumers, other researchers can apply them. ROC trees are one method to accomplish this purpose. ROC trees are one method to accomplish this purpose.