Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,
Published byModified over 4 years ago
Presentation on theme: "Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council,"— Presentation transcript:
Cost-Sensitive Classifier Evaluation Robert Holte Computing Science Dept. University of Alberta Co-author Chris Drummond IIT, National Research Council, Ottawa
Classifiers A classifier assigns an object to one of a predefined set of categories or classes. Examples: –A metal detector either sounds an alarm or stays quiet when someone walks through. –A credit card application is either approved or denied. –A medical test’s outcome is either positive or negative. This talk: only two classes, “positive” and “negative”.
Two Types of Error False negative (“miss”), FN alarm doesn’t sound but person is carrying metal False positive (“false alarm”), FP alarm sounds but person is not carrying metal
2-class Confusion Matrix Reduce the 4 numbers to two rates true positive rate = TP = (#TP)/(#P) false positive rate = FP = (#FP)/(#N) Rates are independent of class ratio* True class Predicted class positivenegative positive (#P)#TP#P - #TP negative (#N)#FP#N - #FP * subject to certain conditions
Assumptions Standard Cost Model –correct classification costs 0 –cost of misclassification depends only on the class, not on the individual example –over a set of examples costs are additive Costs or Class Distributions: –are not known precisely at evaluation time –may vary with time –may depend on where the classifier is deployed True FP and TP do not vary with time or location, and are accurately estimated.
How to Evaluate Performance ? Scalar Measures –Accuracy –Expected cost –Area under the ROC curve Visualization Techniques –ROC curves –Cost Curves
What’s Wrong with Scalars ? A scalar does not tell the whole story. –There are fundamentally two numbers of interest (FP and TP), a single number invariably loses some information. –How are errors distributed across the classes ? –How will each classifier perform in different testing conditions (costs or class ratios other than those measured in the experiment) ? A scalar imposes a linear ordering on classifiers. –what we want is to identify the conditions under which each is better.
What’s Wrong with Scalars ? A table of scalars is just a mass of numbers. –No immediate impact –Poor way to present results in a paper –Equally poor way for an experimenter to analyze results Some scalars (accuracy, expected cost) require precise knowledge of costs and class distributions. –Often these are not known precisely and might vary with time or location of deployment.
Why visualize performance ? Shape of curves more informative than a single number Curve informs about –all possible misclassification costs* –all possible class ratios* –under what conditions C1 outperforms C2 Immediate impact (if done well) * subject to certain conditions
Operating Range Slope indicates the class distributions and misclassification costs for which the classifier is better than always-negative ditto for always-positive
Convex Hull Slope indicates the class distributions and misclassification costs for which the red classifier is the same as the blue one.
Creating an ROC Curve A classifier produces a single ROC point. If the classifier has a “sensitivity” parameter, varying it produces a series of ROC points (confusion matrices). Alternatively, if the classifier is produced by a learning algorithm, a series of ROC points can be generated by varying the class ratio in the training set.
When to switch from C4.5 to IB1 ? What is the performance difference ? When to use the default classifiers ? ROC curves for two classifiers. How to tell if two ROC curves’ difference is statistically significant ?
How to average them? How to compute a confidence interval for the average ROC curve ? ROC curves from two cross-validation runs.
And we would like be able to answer all these questions by visual inspection …
Paired Resampling to Test Statistical Significance Predicted by Classifier1 Predicted by Classifier2 posneg pos3010 neg060 For the 100 test examples in the negative class: FP for classifier1: (30+10)/100 = 0.40 FP for classifier2: (30+0)/100 = 0.30 FP2 – FP1 = -0.10 Resample this matrix 10000 times to get (FP2-FP1) values. Do the same for the matrix based on positive test examples. Plot and take 95% envelope as before.
Paired Resampling to Test Statistical Significance classifier1 classifier2 FN2-FN1 FP2-FP1
Correlation between Classifiers Predicted by Classifier1 Predicted by Classifier2 posneg pos3010 neg060 High Correlation Low Correlation (same FP1 and FP2 as above) Predicted by Classifier1 Predicted by Classifier2 posneg pos040 neg30
Conclusions Scalar performance measures should not be used if costs and class distributions are not exactly known or might vary with time or location. Cost curves enable easy visualization of –Average performance (expected cost) –operating range –confidence intervals on performance –difference in performance and its significance
Fin Cost curve software is available. Contact: email@example.com Thanks to Alberta Ingenuity Centre for Machine Learning (www.aicml.ca)