Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen.

Similar presentations

Presentation on theme: "1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen."— Presentation transcript:

1 1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen

2 2 Introduction to the paper By Samy Bengio, Johnny Mariéthoz and Mikaela Keller, 2005 For machine learning community and researchers ect, who need to compare models. Content of the paper: Introduces ROC curves very briefly. Points out some risks when using ROC curves for comparing different classifying models. Argues that ROC curves can be misleading by showing some results. The authors contributes with a so called “Expected Performance Curve”, and argues why it is better for comparing models. Extends EPC with confidence intervals and statistical difference tests. Concludes the paper summarizing their contribution and by listing strenghts and weaknesses of ROC and EPC. Acknowledgement and references

3 3 Content Motivation Introduce terminology and notation, define problem. Introduce ROC curves Example: how to calculate a ROC Present arguments of why ROC curves should be used with great care Introduce EPC Continue example showing how to calculate an EPC Present arguments of why EPC might be better than ROC Confidence interval My opinion Discussion

4 4 Motivation ROC analysis is an important why to compare binary classifier models. Can be used to select optimal models and discard suboptimal models. Area of use: Medicine (diagnostic testing, evaluate evidence-based medicine approaches) Epidemiology (factors affecting health, evaluate optimal treatment approaches) Radiology (radar signals, evaluate new radiology techniques ) Psychology (signal detection, assess human detection of weak signals) Machine Learning (evaluation of machine learning techniques) …

5 5 Definition of 2-class classifiers Definition of 2-class classification problems: Apply function and associated threshold on a seperate test data set (true class must be known) and count the outcome.

6 6 Confusion matrix Given a 2 class classifier and an instance, there are four possible outcomes: TP: instance is positive and is classified as positive FN: instance is positive and is classified as negative TN: instance is negative and is classified as negative FN: instance is negative and is classified as positive

7 7 Perfomance metrics Selected measure is a pair which is generically called V1 and V2. V1 and V2 can be calculated in many ways depending on the situation. All are simple combinations of TP, TN, FP and FN. Exact calculation of V1 and V2 is not important in this paper.

8 8 Perfomance metrics An unique measure generically called V combines V1 and V2 V can also be calculated in several ways depending on the situation (Half Total Error Rate)

9 9 What is a ROC curve? ROC Abbreviation for ”Receiver Operating Characteristics”. Technique for visualizing, organizing and selecting classifiers based on their performance. ROC can both be presented as a graph or a curve. Classifiers Discrete classifiers (decision trees, rule sets ect.) Probabilistic classifiers (Naive Bayes, neural network ect.) Varying a threshold for a probabilistic classifier will trace a curve (ROC) Following example will show this.

10 10 Example

11 11 Example

12 12 Example Threshold

13 13 Example Threshold

14 14 Example Threshold

15 15 Example Threshold

16 16 Example

17 17 Example

18 18 ROC curves BEP = Breake Even Point BEP corresponds to the threshold nearst to a solutions such that V1 = V2 The selected threshold have a significant impact on the model. The threshold represents the a trade- off between giving importance to V1 or V2.

19 19 Potential risk of using ROC Each point corresponds to a particular setting of the threshold. But in “real applications” the thresholds need to be decided before seeing the test set. Normally the threshold is found by searching for the BEP using some equation. Possibility of mismatch because training set is different from the test set. Situations may occur where the optimal threshold found be using the training set, doesn’t correspond to the optimal threshold on the test set. One parameter, the threshold, is tuned using the training set. Potential risk to expect that the training error reflects the general error. “Real applications often suffer from an additional mismatch between training and test conditions”. Risk of a different trade-off (V1, V2) in test set. ROC curves does not take the risk of a mismatch into account. This probalility should be reflected in the procedure when calculating the performance curve.

20 20 Potential risk of using ROC ROC’s of two real models for a Text-Independent Speaker Verifacation task. Looking at the curves only model B seems to be better than model A. Looking at the thresholds, A is actually the best model.

21 21 Expected performance curve EPC present a range of possible expected performance on the test set. The calculation takes into account the possible mismatch while estimating the desired threshold. A parameter alpha is used to estimate the possible missmatch of the threshold. Framework: Paremetric performance measure: C( V1(θ, D), V2(θ, D);  ) Depends on: The parameter , V1 and V2 computed on some data D for the threshold θ. Example: C( V1(θ, D), V2(θ, D);  ) = C( Precision(θ, D), Recall(θ, D) ;  ) = - (   Precision(θ, D) + (1 -  )  Recall(θ, D)) Procedure: Vary  inside a reasonable range and for each  estimate θ that minimizes C(-,-;  ) on a development set and then use the obtained θ to compute V on the test set. At last plot V with respect to .

22 22 EPC Algorithm

23 23 Example

24 24 Example

25 25 Example

26 26 Example

27 27 Example

28 28 Example of an typical EPC Alpha > 0,5 = more importance to false acceptance errors Alpha < 0,5 = more importance to false rejection errors

29 29 EPC in real applications Expected Performance Curves for person authentication, where one wants to trade- off false acceptance rates with false rejection rates. Expected Performance Curves for text categorization, where one wants to trade- off precision and recall and print the F1 measure.

30 30 Confidence Interval Confidence intervals are used to indicate the reliability of an estimate

31 31 My opinion The authors got a point and the idea is good. Good for comparing models… …but hard to read much from EPC, ROC more informative. Cumbersome to compute EPC. Useful… maybe? Apparently only used by the authors?

32 32 End of Line Questions Discussion

Download ppt "1 The Expected Performance Curve Samy Bengio, Johnny Mariéthoz, Mikaela Keller MI – 25. oktober 2007 Kresten Toftgaard Andersen."

Similar presentations

Ads by Google