1. ROC curve estimation

2. Index Introduction to ROC ROC curve Area under ROC curve Visualization using ROC curve

3. ROC curve Originally stands for Receiver Operating Characteristic curve. It is used widely in biomedical applications like radiology and imaging. An important utility here is to assess classifiers in machine learning.

4. Example situation Consider diagnostic test for a disease Test has 2 possible outcomes: Positive or negative. Now based on this we will explain the various notations used in ROC curves in the next slide.

5. Data distribution available Test Result Pts with disease Pts without the disease

6. Test Result Call these patients “negative”Call these patients “positive” Threshold

7. Test Result Call these patients “negative”Call these patients “positive” without the disease with the disease True Positives Some definitions...

8. Test Result Call these patients “negative”Call these patients “positive” without the disease with the disease False Positives

9. Test Result Call these patients “negative”Call these patients “positive ” without the disease with the disease True negatives

10. Test Result Call these patients “negative”Call these patients “positive” without the disease with the disease False negatives

11. Confusion Matrix Confusion matrix is defined as a matrix consisting of two rows and two columns. The orientation of entries in the confusion matrix is as follows if say the confusion matrix is called CMat. Then CMat[1][1]=True Positives CMat[1][2]=False Positives. Similarly CMat[2][1]=False Negatives and CMat[2][2]=True Negatives.

12. 2-class Confusion Matrix Reduce the 4 numbers to two rates true positive rate = TP = (#TP)/(#P) false positive rate = FP = (#FP)/(#N) Rates are independent of class ratio* True class Predicted class positivenegative positive (#P)#TP#P - #TP negative (#N)#FP#N - #FP

13. Comparing classifiers using Confusion Matrix True Predicted posneg pos6040 neg2080 True Predicted posneg pos7030 neg50 True Predicted posneg pos4060 neg3070 Classifier 1 TP = 0.4 FP = 0.3 Classifier 2 TP = 0.7 FP = 0.5 Classifier 3 TP = 0.6 FP = 0.2

14. Interpretations from the Confusion matrix The following metrics for a classifier can be calculated using the confusion matrix. These can be used for evaluating the classifier. Accuracy = (TP+TN) Precision = TP/(TP+FP) Recall = TP/(TP+FN) F-Score = 2*recall*precision/(recall + precision)

15. True Positive Rate (sensitivity) 0% 100% False Positive Rate (1-specificity) 0% 100% ROC curve

16. True Positive Rate 0%0% 100% False Positive Rate 0%0% 100% True Positive Rate 0%0% 100% False Positive Rate 0%0% 100% A good test: A poor test: ROC curve comparison

17. Area under ROC curve (AUC) Overall measure of test performance Comparisons between two tests based on differences between (estimated) AUC For continuous data, AUC equivalent to Mann- Whitney U-statistic (nonparametric test of difference in location between two populations) Determines the accuracy of a classifier in machine learning.

18. True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % AUC = 50% AUC = 90% AUC = 65% AUC = 100% True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % AUC for ROC curves

19. Further Evaluation methods ROC curve based visualization The visualization of the ROC curve is a very good method of evaluating the classifier. Tools like Matlab, Weka and Orange provide facilities to support visualization of the ROC curve.

20. ROCR is one such tool which provides effective visualization.