2. ECML-20011 Estimating the predictive accuracy of a classifier Hilan Bensusan Alexandros Kalousis

3. Why do we need to estimate the classifier performance?  To perform model selection without a previously established pool of classifiers.  To make meta-learning more automatic and less dependent on human experts.  To gain insight into the area of expertise of different classifiers

4. Meta-learning  Meta-learning is the endeavour to learn something about the expected performance of a classifier from previous applications.  It depends heavily on the way datasets are characterised.  It has concentrated on predicting the suitability of a classifier and on classifier selection from a pool.

5. Regression to predict performance  In this paper we examine an approach to the direct estimation of performances through regression.  The work is somehow related to zooming for ranking but there no knowledge about the classifiers is gained.

6. Previous work includes  João Gama and Pavel Brazdil in work related to StatLog (one dataset characterisation only and poor results reported in NMSE).  So Young Sohn (with StatLog datasets with boosting but better results).  Recent paper by Christian Koepf (good results with few classifiers and artificial datasets only).

7. Our approach  Broaden the research by comparing different dataset characterisation strategies and different regression methods.  A metadataset for each classifier is composed by a set of dataset characterisation attributes and the performance of the classifier in each dataset.

8. We concentrate on 8 classifiers:  two decision tree classifiers (C5.0tree and Ltree),  Naive Bayes  Linear discriminant,  Two rule methods (C5.0rules and ripper),  Nearest neighbor,  And a combination method (C5.0boost)

9. Strategies of dataset characterization:  A set of information-theoretical and statistical features of the datasets developed after StatLog (dct).  A finer grained development of the StatLog characteristics, where histograms are used to describe the distributions of features computed for each attribute of a dataset (histo).  Landmarking (land).

10. ECML-20019 Landmarking  A characterization technique where the performance of simple, bare-bone learners in the dataset is used to characterise it.  In this paper we use seven landmarkers: Decision node, Worst node, Randomly chosen node, Naive Bayes, 1-Nearest Neighbour, Elite 1-Nearest Neighbour and Linear Discriminant.

11. ECML-200110 Regression on accuracies  The quality of the estimate depends on its closeness to the actual accuracy achieved by the classifier, measured by the Mean Absolute Deviation (MAD) using 10 fold xval.  MAD is defined as the sum of the absolute differences between real and predicted values divided by the number of test items.  dMAD, the MAD obtained by predicting always the mean error, is used as reference.

12. ECML-200111 Regression methods and datasets  We used a kernel method and Cubist for regression.  65 datasets from UCI and METAL were used.  Classifier performance is the mean of the accuracy on the 10 xval folds

13. ECML-200112 Estimating with kernel Classifier dct histo land dMAD C5.0boost 0.112 0.123 0.050 0.134 C5.0rules 0.110 0.121 0.051 0.133 C5.0tree 0.110 0.123 0.054 0.137 lindisc 0.118 0.129 0.063 0.137 ltree 0.105 0.113 0.041 0.132 Near.Nei. 0.120 0.138 0.081 0.153 NaiBayes 0.121 0.143 0.064 0.146 ripper 0.113 0.128 0.056 0.145

14. ECML-200113 Estimating with Cubist Classifier dct histo land dMAD C5.0boost 0.103 0.128 0.033 0.134 C5.0rules 0.121 0.126 0.036 0.133 C5.0tree 0.114 0.130 0.044 0.137 lindisc 0.118 0.140 0.054 0.137 ltree 0.114 0.121 0.032 0.132 Near.Nei. 0.150 0.149 0.067 0.153 NaiBayes 0.126 0.149 0.044 0.146 ripper 0.128 0.131 0.041 0.145

15. ECML-200114 Using estimates to rank  Rankings are compared for similarity using Spearman’s rank correlation.  Zooming cannot be applied to land since we should not use a classifier to rank itself (we use land-).  We compare the ranking estimates with the true ranking.  The default ranking is computed over all datasets (C5b,r,t,Lt,rip,nn,nb,lind downwards).

16. ECML-200115 Average Spearman's Correlation Coefficients with the True Ranking RankingsKernelCubistZooming Default 0.330 dct 0.435 0.083 0.341 histo 0.405 0.174 0.371 land 0.180 0.185 - land- - 0.090 0.190

17. ECML-200116 Gaining insight about classifiers Example, a land rule: (34 cases, mean error 0.218) IF Rand_Node <= 0.57 Elite_Node > 0.084 THEN mlcnb = 0.167 + 0.239 Rand_Node - 0.18 Worst_Node + 0.105 Elite_Node

18. ECML-200117 Conclusions  Regression can be used to estimate performances.  Meta-learning needs good dataset characterisation.  Landmarking is the best dataset characterisation strategy for performance estimation but not the best one for ranking.  Future work includes further exploration of dataset characterisation strategies and of the results of combining them (as well as explaining the still odd result of landmarking in ranking).