Download presentation

1
ROC & AUC, LIFT ד"ר אבי רוזנפלד

2
**Introduction to ROC curves**

ROC = Receiver Operating Characteristic Started in electronic signal detection theory (1940s s) Has become very popular in biomedical applications, particularly radiology and imaging גם בשימוש בכריית מידע

3
**False Positives / Negatives**

Confusion matrix 1 Confusion matrix 2 P N 20 10 30 90 P N 10 20 15 105 FN Actual Actual FP Predicted Predicted Precision (P) = 20 / 50 = 0.4 Recall (P) = 20 / 30 = 0.666 F-measure=2*.4*.666/1.0666=.5

4
**Different Cost Measures**

The confusion matrix (easily generalize to multi-class) Machine Learning methods usually minimize FP+FN TPR (True Positive Rate): TP / (TP + FN) = Recall FPR (False Positive Rate): FP / (TN + FP) = Precision Predicted class Yes No Actual class TP: True positive FN: False negative FP: False positive TN: True negative

5
**Specific Example People without disease People with disease**

Test Result

6
**Threshold Test Result Call these patients “negative”**

Call these patients “positive” Test Result

7
**Some definitions ... True Positives Test Result**

Call these patients “negative” Call these patients “positive” True Positives Test Result without the disease with the disease

8
**False Positives Test Result Call these patients “negative”**

Call these patients “positive” False Positives Test Result without the disease with the disease

9
**True negatives Test Result Call these patients “negative”**

Call these patients “positive” True negatives Test Result without the disease with the disease

10
**False negatives Test Result Call these patients “negative”**

Call these patients “positive” False negatives Test Result without the disease with the disease

11
**Moving the Threshold: left**

‘‘-’’ ‘‘+’’ Test Result without the disease Which line has the higher recall of -? Which line has the higher precision of -? with the disease

12
**ROC curve True Positive Rate (Recall)**

0% 100% False Positive Rate (1-specificity) 0% 100%

13
**Figure 5.2 A sample ROC curve.**

Jagged line, shows, instance by instance (in order by sorted list shown (Table 5.6)) increases in true positives (correctly predicted as yes) and false positives (incorrectly predicted as yes) It HAD BETTER BE ABOVE THE DIAGONAL LINE or else the learning method is hurting us! Smooth curves may be drawn – or generated using cross validation … see next slide Figure 5.2 A sample ROC curve.

14
**סוגים שונים של ROC גרפים**

15
**Area under ROC curve (AUC)**

מדד כללי השטח מתחת לגרף ROC 0.50 הוא מחירה רנדומאלי, 1.0 הוא מושלם.

16
**AUC for ROC curves AUC = 100% AUC = 50% AUC = 90% AUC = 65%**

True Positive Rate 0% 100% False Positive Rate True Positive Rate 0% 100% False Positive Rate AUC = 100% AUC = 50% True Positive Rate 0% 100% False Positive Rate True Positive Rate 0% 100% False Positive Rate AUC = 90% AUC = 65%

17
**Lift Charts X axis is sample size: (TP+FP) / N Y axis is TP**

הגדרה פורמאלי: דיוק המודל / דיוק רנדומאלי 80% of responses for 40% of cost Lift factor = 2 Model 40% of responses for 10% of cost Lift factor = 4 Random

18
Lift factor Lift Value Sample Size

19
הקשר בין המדדים

20
בעיית הOVERFITTING

21
**10-fold cross-validation (one example of K-fold cross-validation)**

1. Randomly divide your data into 10 pieces, 1 through k. 2. Treat the 1st tenth of the data as the test dataset. Fit the model to the other nine-tenths of the data (which are now the training data). 3. Apply the model to the test data (e.g., for logistic regression, calculate predicted probabilities of the test observations). 4. Repeat this procedure for all 10 tenths of the data. 5. Calculate statistics of model accuracy and fit (e.g., ROC curves) from the test data only.

22
תמונה

23
ניתוח התוצאות

24
The Kappa Statistic Kappa measures relative improvement over random prediction Dreal / Dperfect = A (accuracy of the real model) Drandom / Dperfect= C (accuracy of a random model) Kappa Statistic = (A-C) / (1-C) = (Dreal / Dperfect – Drandom / Dperfect ) / (1 – Drandom / Dperfect ) Remove Dperfect from all places (Dreal – Drandom) / (Dperfect – Drandom) Kappa = 1 when A = 1 Kappa 0 if prediction is no better than random guessing

25
**Aside: the Kappa statistic**

Two confusion matrix for a 3-class problem: real model (left) vs random model (right) Number of successes: sum of values in diagonal (D) Kappa = (Dreal – Drandom) / (Dperfect – Drandom) (140 – 82) / (200 – 82) = 0.492 Accuracy = 140/200 = 0.70 Predicted Predicted a b c 88 10 2 100 14 40 6 60 18 12 120 20 200 a b c 60 30 10 100 36 18 6 24 12 4 40 120 20 200 total total Actual Actual total total

26
**The kappa statistic – how to calculate Drandom ?**

Expected confusion matrix, E, for a random model Actual confusion matrix, C a b c 88 10 2 100 14 40 6 60 18 12 120 20 200 total a b c ? 100 60 40 120 20 200 total Actual Actual The idea is to compare actual results with what would have happened if a random predictor predicted answers in the same proportion that the actual predictor did On the left is the actual results – with = 140 correct out of 200 (70%) The actual proportions are 100, 60, and 40 for A,B,C The prediction proportions are 120, 60, 20 for A,B, and C On the right. The prediction proportions are matched – but predictions are random so those 120 predictions of A are split among all of the actual values since 50% of the actual answers are A (100 of 200), of those 120 predictions of A we expect half to actually be correct (60) Since 30% of actual answers are B (60 of 200), of the 120 predictions of A, we expect 30% to be for instances that are actually Bs (36) Since 20% of actual answers are C (40 of 200), of the 120 predictions of A, we expect 20% to be for instances that are actually Cs (24) We’re going to have 60 predictions of B, but predictions are random so we expect those predictions to be split among all of the actual values since 50% of the actual answers are A (100 of 200), of those 60 predictions of B we expect half to be for instances that are actually As (30) Since 30% of actual answers are B (60 of 200), of the 60 predictions of B, we expect 30% to be correct (18) Since 20% of actual answers are C (40 of 200), of the 60 predictions of B, we expect 20% to be for instances that are actually Cs (12) We’re going to have 20 predictions of C, but predictions are random so we expect those predictions to be split among all of the actual values since 50% of the actual answers are A (100 of 200), of those 20 predictions of C we expect half to be for instances that are actually As (10) Since 30% of actual answers are B (60 of 200), of the 20 predictions of C, we expect 30% to be for instances that are actually Bs (6) Since 20% of actual answers are C (40 of 200), of the 20 predictions of C, we expect 20% to be correct (4) The expected results with stratified random prediction is =82 So this classifier being tested squeezed an extra = 58 correct predictions The best possible classifier (100% correct) could have obtained an extra =118 correct predictions So our classifier got 58 out of possible 118 extra correct predictions (49.2%) that’s the Kappa Statistic total total 100*120/200 = 60 Rationale: 100 actual values, 120/200 in the predicted class, so random is: 100*120/200

27
לקראת התרגיל...

Similar presentations

© 2019 SlidePlayer.com Inc.

All rights reserved.

To make this website work, we log user data and share it with processors. To use this website, you must agree to our Privacy Policy, including cookie policy.

Ads by Google