Presentation is loading. Please wait.

Presentation is loading. Please wait.

Comparing Classifiers On Comparing Classifiers : Pitfalls to Avoid and Recommended Approach Published by Steven L. Salzberg Department of Computer Science,

Similar presentations


Presentation on theme: "Comparing Classifiers On Comparing Classifiers : Pitfalls to Avoid and Recommended Approach Published by Steven L. Salzberg Department of Computer Science,"— Presentation transcript:

1 Comparing Classifiers On Comparing Classifiers : Pitfalls to Avoid and Recommended Approach Published by Steven L. Salzberg Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA Presenter Jiyeon Kim (April14 th, 2014) 1

2 Introduction How does the researcher choose which classification algorithm to use for a new problem? Comparing the effectiveness of different algorithms on public databases – opportunities or dangers ? Are many comparisons relied on widely shared datasets statistically valid ? 2

3 Contents 1 1 Definitions 2 2 Comparing Algorithms 3 3 Statistical Validity 4 4 Conclusions > > Candidate Questions 3

4 1 Definitions Paired T-Test Hypothesis Testing Significant Level (α) P-value 4

5 1/1 Paired T-Test To determine whether two paired sets differ from each other in a significant way Under this assumption - the paired differences are independent and identically normally distributed 5

6 1/2 Hypothesis Testing Null Hypothesis (H0) vs. Alternative Hypothesis (H1) Reject the null hypothesis (H0), if the p-value is less than the significance level e.g. In the case of Paired T-test, H0 : There is no difference in two populations. H1 : There is a statistically significant difference. 6

7 1/3 Significance Level, α The percentage of the time in which the experimenters make an error Usually, the significance level is chosen to be 0.05 (or equivalently, 5%) A fixed probability of wrongly rejecting the null hypothesis H0, if it is in fact true ( = P(type I error) ) 7

8 1/4 P-Value The probability of obtaining a test statistic result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true Reject the null hypothesis (H0) " when the p-value turns out to be less than a certain significance level, often

9 2 Comparing Algorithms Empirical validation of Classification Research has serious experimental deficiencies Be careful when making conclusion that a new method is significantly better on well-studied datasets 9

10 3 Statistical Validity e.g. Assume that you do 154 experiments ( two-tailed, paired t-test) with significant level You have 154 chances to be significant - The expected number of significant results is 154 * 0.05 = 7.7 Now You have 770 % error rate !! 10

11 3 Statistical Validity Let α* be the error rate of each experiment Then, (1- α*) become the chance that we can get right conclusion If we conduct n independent experiments, the chance of getting them all right is (1- α*) So, the chance that we will make at least one mistake, α is 1 - (1- α*) 11

12 3 Statistical Validity e.g. (This is not correct usage!!) Assume that you do 154 experiments ( two-tailed, paired t-test ) with significant level 0.05 again The significance level for each experiment, α* = 0.05 Then, the right conclusion rate, (1 - α*) = ( ) = 0.95 The chance of getting them all right is ( ) ^154 So, the significance level for all experiments is 1 - (1 - α*) = 1 - (1 – 0.05)^154 = Now You have 99.96% error rate !! 12

13 3 Statistical Validity Then, what should we do? e.g. (This is correct usage!!) α = 1 - (1 – α*)^ in order to obtain results significant at the 0.05 level with 154 results Then, it gives α* which is more stringent than the original significance level 0.05! 13

14 3 Statistical Validity / CAVEAT / This argument is very rough because it assumes that all the experiments are independent of one another! 14

15 * Recommended Tests Simple Binomial Test ANOVA(Analysis of Variance) ( with Bonferroni Adjustment ) 3/1 Alternative statistical tests Statistical Validity / 15

16 To compare two algorithms ( A&B ), a comparison must consider four numbers ; The number of examples that A got right and B got wrong A>B The number of examples that B got right and A got wrong A>B The number that both algorithms got right The number that both algorithms got wrong 3/1 Alternative statistical tests Statistical Validity / 16

17 To compare two algorithms ( A&B ), a comparison must consider four numbers ; The number of examples that A got right and B got wrong A>B The number of examples that B got right and A got wrong A>B simple but much improved way, Binomial Test! 3/1 Alternative statistical tests Statistical Validity / 17

18 3/2 Community Experiments Statistical Validity / Even when using strict significance criteria and the appropriate significance tests, there would be mere accidents of chance In order to deal with this phenomenon, the most helpful resolution is duplication ! 18

19 3/3 Repeated Tuning Statistical Validity / Algorithms are tuned repeatedly on some datasets Whenever tuning takes place, every adjustment should be considered a separate experiment e.g. If 10 tuning experiments were attempted, then significance level should be instead of

20 3/3 Repeated Tuning Statistical Validity / To establish the new algorithms comparative merits, Choose other algorithm that is most similar to the new one to include in the comparison Choose a benchmark data set that illustrates the strengths of the new algorithm Divide the data set into k subsets for cross-validation Run a cross-validation To compare algorithms, use the appropriate statistical test 20

21 3/3 Repeated Tuning Statistical Validity / (A)For each of the k subsets of the data set D, create a training set T = D - k (B) Divide each training set into two smaller subsets, T1 and T2 ; T1 will be used for training, and T2 for tuning (C) Once the parameters are optimized, re-run training on the larger set T (D) Finally, measure accuracy on k (E) Overall accuracy is averaged across all k partitions ; These k values also give an estimate of the variance of the algorithms 21

22 4 Conclusions No single technique is likely to work best on all databases Empirical comparisons should be done for validity of algorithms but these studies must be very careful! - Comparative work should be done in a statistically acceptable framework The contents above are to help experimental researchers steer clear of problems in designing a comparative study. 22

23 > Exam Questions 1 Q) Why should we apply Bonferroni Adjustment to comparing classifiers? 23

24 > Exam Questions 1 A) In case of multiple tests, multiplicity effect occurs if we use same significant level for each test as for all tests. So we need to get more stringent level for each experiment by Bonferroni Adjustment. 24

25 > Exam Questions 2 Q)Assume that you will do 10 experiments for comparing two classification algorithms. Using Bonferroni Adjustment, determine the criterion of α* (the significant level for each experiment) in order to get results that are truly significant at the 0.01 level for 10 tests. 25

26 > Exam Questions 2 A) α = 1 - (1 - α*)^10 = 1 - (1 - α*)^ (1 - α*)^ α* α*

27 > Exam Questions 3 Q) Specify the difference between paired t-test and simple binomial test in comparing two algorithms. 27

28 > Exam Questions 3 A) Paired t-test : determine whether the difference between two algorithms exists or not Binomial test : compare the percentage of times algorithm A > algorithm B versus A < B, with throwing out the ties 28

29 Thank You.. 29


Download ppt "Comparing Classifiers On Comparing Classifiers : Pitfalls to Avoid and Recommended Approach Published by Steven L. Salzberg Department of Computer Science,"

Similar presentations


Ads by Google