1. 1 ASSESSING THE PERFORMANCE OF MEDICAL DIAGNOSTIC SYSTEMS: THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE JOSEPH GEORGE CALDWELL, PH.D. 27 FEBRUARY 2007

2. 2 BACKGROUND TWO-ALTERNATIVE STATISTICAL DECISION PROBLEMS: HOW TO ASSESS ACCURACY? FROM A SERIES OF TEST RESULTS (E.G., BLOOD TESTS, X-RAYS, MRI IMAGES), MUST DECIDE WHETHER SOMEONE HAS A PARTICULAR DISEASE OR CONDITION STATISTICAL CONSULTING REPORT: “DESCRIPTION OF THE STATISTICAL SUBSYSTEM OF THE AUTOMATED RECEIVER OPERATING CHARACTERISTIC (ROC) SYSTEM”

3. 3 THE TWO-ALTERNATIVE STATISTICAL DECISION PROBLEM THE STIMULUS-RESPONSE MATRIX: TYPE 2 ERROR TYPE 1 ERROR

4. 4 HOW TO REPRESENT THE ACCURACY OF THE DECISION PROBLEM? BY VARYING THE DECISION CRITERION, THE PROBABILITIES OF THE TWO TYPES OF ERROR MAY BE ADJUSTED -- A TRADE-OFF BY CHANGING THE DIAGNOSTIC PROCEDURES (E.G., ADDING TESTS, IMPROVING TESTS), THE PROBABILITIES OF BOTH TYPES OF ERRORS MAY BE REDUCED A PROBLEM: TOO MANY STIMULUS-RESPONSE MATRICES. HOW TO SUMMARIZE ACCURACY MORE SUCCINCTLY?

5. 5 THREE EXAMPLES

6. 6 THE RECEIVER OPERATING CHARACTERISTIC (ROC) CURVE

7. 7 AN EXAMPLE

8. 8 AN EXAMPLE (CONT.)

9. 9

10. 10 EXAMPLES OF ROC CURVES

11. 11 APPLICATION TO LOGISTIC REGRESSION ESTIMATE DECISION CRITERION: DECIDE “YES” IF p est > c AND “NO” IF p est <= c VARY c AND CALCULATE THE PROPORTION OF HITS AND FALSE POSITIVES, AND PLOT ON A ROC GRAPH (EACH VALUE OF c YIELDS A DIFFERENT POINT ON THE CURVE). LOGISTIC REGRESSION MODEL:

12. 12 ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

13. 13 ROC CURVES FOR LOGISTIC REGRESSION MODELS (SAS OUTPUT)

14. 14 REFERENCES

15. 15 REFERENCES

16. 16 REFERENCES 1. Swets, John A., and Ronald M. Pickett, Evaluation of Diagnostic Systems: Methods from Signal Detection Theory, Academic Press, New York, 1982 2. Green, David M. and John. A Swets, Signal Detection Theory and Psychophysics, Wiley, 1966, reprinted by Peninsula Publishers, Los Altos, CA, 1988 3. Dorfman, Donald D. and Edward Alf, Jr., "Maximum-Likelihood Estimation of Parameters of Signal- Detection Theory and Determination of Confidence Intervals -- Rating Method Data", Journal of Mathematical Psychology, vol. 6, 1969, pp. 487-496 4. Ogilvie, John C. and C. Douglas Creelman, "Maximum-Likelihood Estimation of Receiver Operating Characteristic Curve Parameters," Journal of Mathematical Psychology, vol. 5, 1968, pp. 377-391. 5. Rau, C. R., Linear Statistical Inference and Its Applications, Wiley, New York, 1965, pp. 302-309 6. Ralston, A., A First Course in Numerical Analysis, McGraw Hill, New York, 1965, p. 348 7. Caldwell, J. G., Description of the Statistical Subsystem of the Automated Receiver Operating Characteristic System, 1995 8. Logistic Regression Examples Using the SAS System, SAS Institute, Cary, NC, Version 6, First Editioin, 1995 9. Bamber, D. (1975), “The Area Above the Ordinal Dominance Graph and the Area Below the Receiver Operating Characteristic Graph,” Journal of Mathematical Psychology, 12, 387-415. 10. Hanley, J. A. and McNeil, B. J. (1982), “The Meaning and Use of the Area Under a Receiver Operating Characteristic (ROC) Curve,” Radiology, 143, 29-36.